[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Thu, 11 Apr 2024 22:31:16 -0500
From: Jacob Bachmeyer <jcb62281@...il.com>
To: Alejandro Colomar <alx@...nel.org>
CC: oss-security@...ts.openwall.com, Sam James <sam@...too.org>,
Joey Hess <id@...yh.name>,
Jonathan Nieder <jrnieder@...il.com>, Andres Freund <andres@...razel.de>,
Lasse Collin <lasse.collin@...aani.org>,
xz@...aani.org
Subject: Re: Analysis on who is Jia Tan, and who he could work
for, reading xz.git
Alejandro Colomar wrote:
> [...]
> On Wed, Apr 10, 2024 at 10:26:13PM -0500, Jacob Bachmeyer wrote:
>
>>> [...]
>>>
>> First, a factual correction: The hypothesis that "Jia Tan" was actually in
>> UTC+03 seems to have been backwards, since the peak activity overlaps only
>> partially with office hours in UTC+03, but does indeed start with 9AM in
>> *UTC-03* by my reckoning. The only problem is that UTC-01 through UTC-03
>> cover various islands in the Atlantic Ocean and a few Eastern parts of South
>> America. All of these strike me as unlikely sockmaster bases. The problem
>> with time zones east of UTC is the observed UTC 17:00 "quitting time" (more
>> below) which only gets /later/ in the local day as you move east.
>>
>> [...]
>>
>
> The timezone can be faked, but it still has implications.
>
Simply setting a different timezone is much easier and cheaper than
faking working hours.
> [...]
>
> For a random timezone, you use either UTC, or the one in a country that
> would be unsuspicious.
... or a timezone that includes a country that everyone will suspect,
like Israel, thus taking heat off of the actual perpetrator.
> [...]
>
> Now, for the false flag. This attack was very likely something that
> they didn't intend to be discovered, as it would be significantly more
> valuable if undiscovered. So we can assume they didn't put those 4
> commits for us to find them.
Competent intelligence services have backup plans and misdirections laid
out in advance. Of course, competent intelligence services also QA
their malware to avoid getting caught the way that the "Jia Tan" gang
did, so competence cannot be fully assumed here.
> [...] might have left a few other traces (for
> example, when git pushing; maybe the logs have an IP).
>
"Jia Tan" is known to have used a Singaporean VPN at Libera Chat. There
is little reason to expect that Git server logs would show other
addresses. Connecting to a VPN and changing timezone are different
actions, and network policies could easily have been set up to block
direct access to GitHub to avoid leaks there.
> [...]
>
> In any of those cases, the timezone is useful information, I think, and
> shouldn't be discarded easily.
>
The problem is that we have no reason to believe that any of those cases
are so. While +0200/+0300 does include a country that I believe to be a
likely suspect, the fact remains that timezones can be trivially faked
and the "working hours" pattern suggests that /most/ of the day-to-day
work of maintaining the sockpuppet was done in another country, one
where the perpetrator /might/ actually face justice.
There is still a possibility that partitioning the commit set might
reveal a second set of working hours, consistent with a group spanning
timezones.
>> There seems to be no 24 hour period where "Jia" made no commits, and what I
>> think is Friday night into Saturday (therefore the Jewish Sabbath) is one of
>> the more frequent late-night periods, while "Jia" seemingly (mostly) took
>> Sundays off. I have read reports where activities were attributed to Israel
>> and two of the key arguments were that APT group did /nothing/ on Friday
>> evenings or Saturdays, and Sunday seemed to be an ordinary work day for
>> them. These characteristics do /not/ describe the "Jia" crew. Whoever
>> "Jia" is, an observant Jew he is not.
>>
>
> That's a very interesting observation.
>
Thank you.
>> I have been looking at this from a different angle, assuming that all of the
>> time zone information in the commits is bogus and looking for patterns in
>> the commit epoch timestamps, which are harder to convincingly fake. The
>> attached "collect.sh" is intended to run in a directory next to a copy of
>> the repository as "xz-backdoored" and extracts the commit and author
>> timestamps in epoch time, further decomposing them into week/time-of-week
>> and day/time-of-day for analysis and plotting. The week and day numbers are
>> counted from 1 Jan 1970, which was a Thursday, so the time-of-week numbers
>> in the output of the attached script are seconds from midnight Thursday. An
>> epoch day number X can be converted back to a date with `date --date='1 Jan
>> 1970 UTC + X days'` and an analogous command converts week numbers to
>> Thursdays. This is a work in progress and I am not yet fully confident that
>> I have correct analysis, in part because my results are different from what
>> others had found before I started, so I am presenting the data extraction
>> script for others to either find problems with or replicate my results. The
>> script was run on a repository clone with master checked out at commit
>> f9cf4c05edd14dedfe63833f8ccbe41b55823b00.
>>
>
> How did you plot them? Do you have a gnuplot script handy or something?
>
I used gnuplot, but the actual plotting scripts add a bunch of
annotations and I do not want to risk contaminating others' analyses
with my own unstated (and perhaps unrecognized) assumptions. I want to
let the data speak, rather than merely give voice to my own prejudices.
I really want other people to do their own analysis using tools of their
choice and either replicate or refute my work here.
That said, since you have asked, the basic setup for a weekly plot I
used is:
8<------
set style data points
set grid xtics ytics mxtics
UTp(days, hours) = (days*86400+hours*3600)
set xrange [0:604800]
set xtics border mirror \
("UTC-Thu" UTp(0,0), "" UTp(0,12) 1, "" UTp(0,17) 1, \
"UTC-Fri" UTp(1,0), "" UTp(1,12) 1, "" UTp(1,17) 1, \
"UTC-Sat" UTp(2,0), "" UTp(2,12) 1, "" UTp(2,17) 1, \
"UTC-Sun" UTp(3,0), "" UTp(3,12) 1, "" UTp(3,17) 1, \
"UTC-Mon" UTp(4,0), "" UTp(4,12) 1, "" UTp(4,17) 1, \
"UTC-Tue" UTp(5,0), "" UTp(5,12) 1, "" UTp(5,17) 1, \
"UTC-Wed" UTp(6,0), "" UTp(6,12) 1, "" UTp(6,17) 1 )
set key on horizontal box
8<------
The weekly plot has minor tics (and extra grid lines) at UTC noon and
UTC 17:00 each day. Note that the week starts on Thursday, because
these are epoch weeks since Thursday, 1 Jan 1970.
Similarly, the basic setup for a daily plot I used is:
8<------
set style data points
set grid
set xrange [0:86400]
set xtics border mirror \
("UTC:00" ( 0*3600), " " ( 1*3600), "02" ( 2*3600), "03" ( 3*3600), \
"04" ( 4*3600), "05" ( 5*3600), "06" ( 6*3600), "07" ( 7*3600), \
"08" ( 8*3600), "09" ( 9*3600), "10" (10*3600), "11" (11*3600), \
"12" (12*3600), "13" (13*3600), "14" (14*3600), "15" (15*3600), \
"16" (16*3600), "17" (17*3600), "18" (18*3600), "19" (19*3600), \
"20" (20*3600), "21" (21*3600), "22" (22*3600), "23" (23*3600) )
set key on horizontal box
8<------
The "collect.sh" attached to my previous message (and duplicated on this
message because it is small and I seem to remember that messages are
supposed to be self-contained on this list) generates one set of files
partitioned by declared author ("Jia Tan", Lasse Collin, all others) and
containing author timestamps and one set partitioned by committer and
containing committer timestamps. Plot the files "using 3:2" for a
weekly plot or "using 5:4" for a daily plot. If using tools that can
associate string data with each point, column 6 contains the abbreviated
commit hash.
>> There is a noticeable cluster in the plot, and about 85% of "Jia Tan"'s
>> commits were in the five hours starting at UTC noon. If we exclude 2024,
>> which seems to have been "crunch time" on getting the backdoor out, that
>> jumps to about 91%. I believe that this pattern *might* be a good indicator
>> for the sock farm containing "Jia Tan" but there are likely to be false
>> positives, so it is probably a weak indicator. Combining this pattern with
>> a claimed timezone (like "Jia"'s UTC+08) where that period is into the night
>> might work better. In UTC+08, that period is 8PM to 1AM, which are unlikely
>> office hours. The peak also ends almost as abruptly as it begins,
>> suggesting that UTC 17:00 was "quitting time" at "Jia"'s office, but that
>> "Jia" did occasionally work late. The five hour active period is consistent
>> with morning planning meetings, followed by general work keeping up "Jia"'s
>> appearances, with a floating lunch break somewhere. Think "rogue state
>> bureaucracy" here.
>>
>
> Hmmm.
>
Having had a little more time to think about this, while "Jia"'s working
hours alone are likely to be a weak indicator, I believe that it may be
useful /if/ the "Jia" gang had aims on any other packages. Add
"pressure to accept the suspect contributer" and you probably have a
fairly good /modus operandi/ by which the "Jia" gang can be recognized.
I also doubt that the "Jia" gang attempted to hit any other packages.
This looks to me like a bunch of bespoke work to infiltrate liblzma in
order to get a backdoor into the sshd process on common distributions.
Reviewing the Git history suggests to me that the entire
CLMUL-accelerated CRC implementation, while itself legitimate, was
contributed to set up an excuse to use ifuncs in liblzma. Tentatively,
I believe that the backdoor could have also used ordinary ELF
constructor functions (I still need to check this), so the use of ifunc
was actually a covert flag to allow the backdoor to be disabled in
environments that were likely to find it, like oss-fuzz, and explain the
difference in configuration for security testing away as due to
limitations of the current GNU libc ifunc implementation.
>> [...]
>>
>> This thread landed in my inbox as I was planning to start work on further
>> partitioning the "Jia Tan" commits, initially by keywords in the commit
>> message. Do commits involving "ifunc" stand out in time from all others?
>> Alejandro's work raises another question: Does time-of-commit correlate to
>> diff size? Alternately: Was the more complex work seemingly done in a
>> different time zone?
>>
>
> I'll try to investigate this. Please update when you finish your study.
> I'm interested.
Thank you. I doubt that the study will be ever entirely finished unless
a whistleblower goes public, but I will keep up with intermediate
results as I find them. Also, thanks for also investigating---the more
my results can be replicated or refuted, the better.
-- Jacob
Download attachment "collect.sh" of type "application/x-sh" (987 bytes)
Powered by blists - more mailing lists
Please check out the
Open Source Software Security Wiki, which is counterpart to this
mailing list.
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
