News for hackers. Stuff that matters.
Re:Reflections on Distrusting xz by irdc on Wednesday April 03, 2024 @06:02AM (#39915426)
One thing that comes to mind is that “Jia Tan” might be more accurately seen as a “sleeper” of some sort: a foot soldier who infiltrates a juicy open source project and waits for further instructions; backdooring sshd might not have been part of the original plan.

Which raises the concerning question of how much more sleeper maintainers there are.

Re:Reflections on Distrusting xz by constantcrying on Wednesday April 03, 2024 @06:51AM (#39915758)
>Which raises the concerning question of how much more sleeper maintainers there are.

Given how easy the infiltration is and how extremely hard to detect it is, likely a lot.

For an intelligence operation it is also extremely cheap, you just need a few knowledgeable developers spending some time each week on the project. The upside being a backdoor into a significant portion of infrastructure, the downside being wasted time.

I do not think it is unlikely that in many important open source software projects there are one or two people assign to keep an eye on things. They don't even need to be malicious, just being somewhat trusted contributors is enough. I would be extremely surprised if the NSA hasn't a couple of guys who keep watch on the Linux Kernel.

Re:Reflections on Distrusting xz by moritonal on Wednesday April 03, 2024 @07:00AM (#39915834)
The irony is that this would make a oddly effective way of having paid open source devs who for the most part just honestly improve projects. With the massive downside they undermine it at a critical moment.
Re:Reflections on Distrusting xz by rightbyte on Wednesday April 03, 2024 @07:52AM (#39916241)
Ye then maybe thanks to them finally we'll have The year of the air gapped Linux desktop.

I dunno what to do if e.g. Debian gets compromized, as in, I can't trust the collective of maintainers.

I assume any Windows machine is backdoored. Trivially proven by forced auto updates.

Maybe air gapping some home computer for sensitive data might be a good idea.

Re:Reflections on Distrusting xz by pants2 on Friday April 05, 2024 @12:13AM (#39938614)
The ideal situation is having multiple intelligence agencies all working on one project and spotting each others' backdoors, so at the end of the day we just have a really secure and well-maintained project.
Re:Reflections on Distrusting xz by david_draco on Wednesday April 03, 2024 @08:46AM (#39916738)
> Given how easy the infiltration is and how extremely hard to detect it is, likely a lot.

I read it exactly the other way around: the infiltration took years and detecting it was the default with a fuzzer, which had to be disabled for the exploit to succeed.

It speaks to the hardening and that hardening should be required more. And of course the precarious roles of maintainers, which have been discussed elsewhere.

Re:Reflections on Distrusting xz by dullcrisp on Wednesday April 03, 2024 @11:03AM (#39918419)
The exploit could be detected with a fuzzer perhaps. The infiltration seems like it was something that would be very easy for a funded intelligence agency to do, and would be nigh undetectable if they had been subtly introducing bugs rather than shipping a sophisticated backdoor to every Linux distro.

You have to assume if this was an intelligence agency they didn’t burn their only agent like this.

Re:Reflections on Distrusting xz by HankB99 on Wednesday April 03, 2024 @02:20PM (#39921031)
> they didn’t burn their only agent like this.

I think I'd characterize that as "their only identity." What's the chance that some number (>1) of actual agents were sharing this identity? If that's the case, I'd extrapolate that to multiple identities. In fact the social engineering to gain the trust of the original maintainer likely involved several identities.

I suspect that detectability of intentionally injected bugs would be very low.

Re:Reflections on Distrusting xz by Thorrez on Wednesday April 03, 2024 @11:56PM (#39926374)
>a fuzzer, which had to be disabled for the exploit to succeed.

According to this comment, the fuzzer wouldn't have detected it. It wasn't necessary to disable the fuzzer:

>https://news.ycombinator.com/item?id=39911249

Re:Reflections on Distrusting xz by geggo98 on Thursday April 04, 2024 @12:38AM (#39926587)
You are right, it took quite some time. On the other hand, it looks like the legitimate part of contributing to xz was only a part time job for the attacker. The rest of the time, they either worked on the exploits, or in other things, like infiltrating other projects using a different handle.

Basically I can imagine the attackers being a well organized group, using work sharing and pipelining. Some members of the group would be preparing exploits, some would infiltrate projects and some would make sure not to get caught. And since infiltrating takes time, they would make sure to have multiple projects in the pipeline, seine in the early contributor stage, some in the social pressure stage, and some in the exploiting stage.

Re:Reflections on Distrusting xz by jannes on Wednesday April 03, 2024 @09:03AM (#39916898)
At this point I wouldn’t be surprised if the NSA also has a couple of Microsoft employees on their payroll.
Re:Reflections on Distrusting xz by cactusfrog on Wednesday April 03, 2024 @09:49AM (#39917445)
I believe LLMs could be useful here as a component of pre-commit hook.
Re:Reflections on Distrusting xz by Gormo on Wednesday April 03, 2024 @10:00AM (#39917614)
> Given how easy the infiltration is and how extremely hard to detect it is, likely a lot.

This particular case seems to be an example of the exact opposite. It took "Jia Tan" two years to conduct all of the social engineering necessary to get into a position to introduce the backdoor, then upon doing so, was caught almost immediately, with the initial discovery not even coming from a dedicated security researcher, but from a sysadmin who kept digging when he saw unusual performance issues.

And the threat actor here deliberately went after the weakest link in the chain. Assuming that the goal was to compromise sshd, the antagonist evidently found it too much of a challenge to attempt to infiltrate OpenSSH itself, so targeted a small-team compression project that only some deployments even link to, and still got caught extremely rapidly.

Re:Reflections on Distrusting xz by constantcrying on Wednesday April 03, 2024 @11:11AM (#39918515)
I don't think you are looking at it the right way. Two years matter if you are a hobbyist or your goal is to compromise the system for some individual gain.

For a nation state actor two years is nothing. Likely the entire attack didn't cost more than a couple of thousand hours of developer time. I would guess it was easily cheaper than 100k in financial terms, that is extremely efficient for an intelligence operation with the upside being access to a large amount of servers.

The method by which he got caught also depended largely upon random chance, some major "if's" needed to happen, the performance reduction was an unfortunate side effect from the perspective of the attacker, really a minor mistake exposed him. Even if he were exposed a couple months later the damage would have been enormous, if that version had become a stable part of any major distro for the next few years hundreds of thousands of machines would have been vulnerable.

There is absolutely no reason to assume that if another attack of this quality happens it will not find it's way into some stable distro.

Re:Reflections on Distrusting xz by Gormo on Wednesday April 03, 2024 @11:45AM (#39918970)
> For a nation state actor two years is nothing.

Perhaps it is, perhaps not. If they are playing a long game, two years might be a straightforward investment. But the point is that it still took two years either way -- the fact that something is only feasible for big institutional actors implies that it is not in itself "easy".

And the kinds of organizations that have the motivation and resources to engage in this kind of long-game infiltration by definition have the resources to influence, infiltrate, and manipulate in a variety of ways. It is absolutely not clear here whether having formal organizations involved in directing the development of xz would have made it easier or harder for whomever was behind this attack.

> The method by which he got caught also depended largely upon random chance,

Why do you presume it's random chance, and that it wouldn't be better represented by more complex probability models? On the surface, it seems like the chance of overall detection would be reasonably represented as an equivalent of an MTBF calculation, with the number of skilled users capable of detecting the vulnerability through normal use as an input. (Not to say that we have enough data to calculate this probability, or that this necessarily even happens enough to have a statistically significant sample -- it might all be "black swans" -- but there's certainly non-random stochastic inputs into this.)

> There is absolutely no reason to assume that if another attack of this quality happens it will not find it's way into some stable distro.

There's no guarantee for any of this. All rules can be gamed, all incentives can be manipulated, all organizations can be compromised. There is no perfect solution to security, and no guarantee that any prescriptive solutions based on generalizing from the specifics of one incident would not create more opportunities for attackers rather than fewer.

Re:Reflections on Distrusting xz by constantcrying on Wednesday April 03, 2024 @12:32PM (#39919525)
I don't disagree, I am not even sure what exactly you are arguing against.

I completely agree with your comment about the detection being the result of iterative low probability chances of detection. My point was that, even if that is the case, there was a major chance that it would have gone into many stable distos or would have been exploited.

Re:Reflections on Distrusting xz by Gormo on Wednesday April 03, 2024 @01:23PM (#39920302)
> I don't disagree, I am not even sure what exactly you are arguing against.

I am arguing that this incident demonstrated the resilience of the FOSS model to even extremely strategic, long-term compromise attempts. Some antagonist invests years worth of time, effort, and money to introduce this backdoor, and the openness of the entire process allowed a single engineer to unravel the whole thing almost immediately after the backdoor was introduced, and every distro immediately sprung into action and neutralized it entirely.

All of the people complaining about the vulnerability of small projects, the pseudonimity of contributors, the need for more institutional involvement, etc. are getting things exactly backwards.

Imagine if this were a closed-source project funded and managed by an opaque institution, and the attacker used a different set of social engineering tactics to backdoor the code from within the org. Suppose then that Andreas Freund noticed the exact same behavior and set out to investigate it. How far would he have gotten?

This incident validates the "many eyes" concept and is a point in favor of the FOSS model.

Re:Reflections on Distrusting xz by kevans91 on Wednesday April 03, 2024 @01:29PM (#39920385)
fwiw, characterizing Andres as a sysadmin isn't really the whole picture; he's a postgres developer that conducts benchmarking operations with some frequency (and he's quite good at what he does)... he's perhaps naturally a bit more sensitive to things like the cumulative effect of 500ms or so over a number of sshd invocations.
Re:Reflections on Distrusting xz by Gormo on Friday April 05, 2024 @11:30AM (#39943624)
You're right -- I went back and changed "sysadmin" to "engineer". Either way, though, he was not a dedicated security researcher, and managed to unravel this entire thing upon noticing an anomaly in the course of his regular work.
Re:Reflections on Distrusting xz by berkes on Wednesday April 03, 2024 @06:30AM (#39915603)
I once got a (probably scam) offer for adding a cryptominer to a library that I maintained at that time. And a more serious offer to add trackers to a popular >1M installs app.

Both cases I obviously ignored it. But it made me aware of a nasty attack vector: someone who's thanklessly building a wordpress-plugin, pip, npm, or whatever software, thanklessly dealing with issues, PRs, support, maintainence, often for no pay, suddenly gets offered three figure sums to add a few lines of "affiliate stuff" or such. There are many places in the world, or people in situations where this amount of money really makes a compelling case.

Re:Reflections on Distrusting xz by irdc on Wednesday April 03, 2024 @06:40AM (#39915671)
“Given enough underfunded maintainers, all security is shallow.”[0]

0. https://en.wikipedia.org/wiki/Linus%27s_law

Re:Reflections on Distrusting xz by lenerdenator on Wednesday April 03, 2024 @10:00AM (#39917606)
I wouldn't say "funding" is necessarily the problem.

Most maintainers do it because they like doing it. Their main limiting factor is time. I can drop a million dollars an hour into a maintainer's lap; that doesn't mean they can dedicate every waking moment to a project. They still have human needs that money can't buy like sleep, family obligations, and health concerns. And that's making the assumption that the maintainer uses that million/hr to quit their job.

No, the problem is a lack of trustworthy candidates for maintainership and a lack of time. There are components of a GNU userland that are now too complex for a single human to both maintain and enhance at the same time. We now need to target multiple distros (really, more than are necessary, strictly speaking) and ISAs. Most are written in systems programming languages like C that are more complex than the average software engineer in 2024 works with.

We need consolidation, simplification, maintainer redundancy, and a trust/governance framework for packages.

Re:Reflections on Distrusting xz by mikrotikker on Thursday April 04, 2024 @08:36AM (#39929619)
We need to utilise a specialised AI to scan through the code looking for bugs and security holes. Imagine if openai donated server time to this.
Re:Reflections on Distrusting xz by strogonoff on Wednesday April 03, 2024 @07:36AM (#39916098)
I believe the problem of thankless maintenance is best solved with two things: the thanks (yes, we are all human and want recognition and appreciation from fellow humans)[0], and a stable employment (work for a good large business while open-sourcing what’s possible)[1].

If you do OSS for profit, then it only becomes a question of where is more money; but for someone who works a reliable job with insurance, relationships and other implications then the stakes may be a bit different.

Many of the biggest OSS projects today were started by people who had no money in mind whatsoever; once we feel relatively secure we are motivated by our innate desire to tinker, create cool things and show it off.

[0] Undermined by LLMs that are used to gobble up your code and suggest it to others commercially and without attribution.

[1] Undermined by low employment protections (if you can expect to be fired at any time, you would be less loyal), and by LLMs (whatever you open-source now more directly benefits Microsoft or whatever).

Re:Reflections on Distrusting xz by david_allison on Wednesday April 03, 2024 @07:49AM (#39916207)
3. More maintainers

Days of 100+ notifications aren't easy. Things will slip through

Re:Reflections on Distrusting xz by strogonoff on Wednesday April 03, 2024 @07:49AM (#39916216)
Agreed, but especially in light of recent events it’d be important to know who they are, and that’s not always easy.
Re:Reflections on Distrusting xz by aleph_minus_one on Wednesday April 03, 2024 @08:51AM (#39916780)
> and a stable employment (work for a good large business while open-sourcing what’s possible)

Even if it was hypothetically possible to open-source basically everything that the team in which I work produces:

The software that I work on is very specialized software that is used by the company's employees and customers for specialized purposes. Imagine some nice LoB application that is actually somewhat comfortable to use. It basically does "what the users need" and is thus deeply ingrained in some parts of the company's workflows. The only use someone outside the industry might have for it is "cosplaying being employed in this industry".

A lot of software that is developed (in particular in companies that don't sell or rent software) is of this kind.

Thus: the open-source scene does in my opinion not have any use for a huge amount of software that is actually developed and actively used.

Re:Reflections on Distrusting xz by strogonoff on Wednesday April 03, 2024 @09:01AM (#39916886)
> The software that I work on is very specialized software that is used by the company's employees and customers for specialized purposes

Oh really? Welcome to the club. Our very specialized software for very specialized purposes used Django with a certain auth provider. So I refactored that into a standalone Django app that painlessly handles this specific OAuth provider, configurable via settings with sane defaults, and open-sourced it. (The refactoring was very beneficial to myself, that part of the project got instantly nicer to work with.)

Of course, it is small beans compared to something algorithmically hardcore (I was a junior myself back then), but it’s just an example.

Any software, no matter how specialized and bespoke, can be expressed as many self-contained isolated components that individually know nothing about that specialization and bespokeness. In fact, such factoring is generally a sign of good design: you may have heard of the loose coupling & high cohesion principle—once you follow it, open-sourcing a particular component is very straightforward.

Note, though, that if your contract has certain licensing provisions in certain countries you may not be allowed to unilaterally open-source anything during the full term of employment (even if it is unrelated to your dayjob). You may need to get approval first. However, many good tech companies are reasonable when it comes to open-sourcing non-core components.

Re:Reflections on Distrusting xz by david_allison on Wednesday April 03, 2024 @07:32AM (#39916062)
> suddenly gets offered three figure sums to add a few lines of "affiliate stuff" or such

Back of the envelope calculation: you're looking at 2 orders of magnitude more money from "affiliate stuff" than you would be from generous user donations

Re:Reflections on Distrusting xz by berkes on Wednesday April 03, 2024 @03:26PM (#39921888)
Well, yes. But it's also something you can do once. When (not if) it comes out, all credibility is lost.

Whereas donations, regardless of how puny, are recurring and potentially forever.

Re:Reflections on Distrusting xz by acdha on Wednesday April 03, 2024 @07:38AM (#39916119)
> There are many places in the world, or people in situations where this amount of money really makes a compelling case.

It’s especially easy to imagine that using the classic intelligence agency playbook: monitor high-impact maintainers and look for leverage before making the approach (“hey, saw your post about the divorce settlement and that $%#@ cleaning you out. My affiliate marketing pays in bitcoin…”) just as they’ve done for ages.

Re:Reflections on Distrusting xz by moritonal on Wednesday April 03, 2024 @07:02AM (#39915851)
It's why I actually always encourage app devs to charge for their apps, even open source ones. It creates an exchange of value for the author to feel valued and detract from these vectors.
Re:Reflections on Distrusting xz by kijin on Wednesday April 03, 2024 @07:29AM (#39916041)
This is what worries me more.

It's easy to point a finger at a specific Bad Guy® and shout "He did it!" It's much harder to face the reality that any maintainer of any open-source project can slowly burn out to a point where they become accomplices in an attack, or at least turn a blind eye.

The pool of open-source developers does not split cleanly into honest contributors and evil agents. The boundary is quite fluid -- more so in some circles than in others -- and there are always temptations to move from one side to the other and back again.

Re:Reflections on Distrusting xz by xign on Wednesday April 03, 2024 @09:41PM (#39925634)
That's definitely true. And a lot of times it could be a random open source project that is under the radar and rarely thought about. E.g. The Great Suspender Chrome extension which was sold to an unknown buyer which later turned it to malware: https://www.bleepingcomputer.com/news/security/the-great-sus...
Re:Reflections on Distrusting xz by andrewinardeer on Wednesday April 03, 2024 @06:33AM (#39915630)
I believe this is a nation state actor and there are a a fleet of 'Jia Tans' working on other OSS projects to backdoor operating systems.

And some have probably succeeded.

Re:Reflections on Distrusting xz by prmoustache on Wednesday April 03, 2024 @07:02AM (#39915852)
I wouldn't limit that to OSS projects. How many of them managed to get hired and are working for Microsoft, Apple, Google, Oracle or Amazon?

In some cases they don't even need to introduce backdoors themselves but just review and spot bugs they don't correct or raise issues for but communicate to mothership. They could even work in team with having one building the backdoor and the other approving the code.

Most companies have more thorough processes to avoid this but that doesn't mean those processes are applied correctly everytime, especially if more than one malicious engineer is involved.

Re:Reflections on Distrusting xz by adql on Wednesday April 03, 2024 @07:15AM (#39915938)
I'm now imagining a department where every single worker is a spy for different government and they play endless game of "add exploit, close off the other people's exploits". And all of them think they are the 10x developer because all the other people do in their view is pushing shoddy code
Re:Reflections on Distrusting xz by simonvc on Wednesday April 03, 2024 @07:26AM (#39916023)
I worked on a project just like this once.. a mobile phone network build in the middle east before the arab spring. 10/10 would not repeat the experience.
Re:Reflections on Distrusting xz by belorn on Wednesday April 03, 2024 @07:37AM (#39916106)
Most of the time they don't need infiltrators. Governments can just pressure companies with export controls or warrantless surveillance to get backdoors into commercial systems. OSS projects require different methods because the more direct method would be discarded by the community and forked.
Re:Reflections on Distrusting xz by StayTrue on Wednesday April 03, 2024 @09:09AM (#39916986)
> Most of the time they don't need infiltrators. Governments can just pressure companies with export controls or warrantless surveillance to get backdoors into commercial systems.

Or they simply pay companies with a “support contract” in return for embedding spyware that sells out customers. Seen that first hand (private key exfiltration), resigned the same day.

Lots of comments saying we need to do something about the OSS supply chain but in my estimation the problem is much worse with closed source commercial software.

Re:Reflections on Distrusting xz by autoexec on Wednesday April 03, 2024 @07:54AM (#39916264)
> I wouldn't limit that to OSS projects. How many of them managed to get hired and are working for Microsoft, Apple, Google, Oracle or Amazon?

They don't have to sneak into those companies tho, they just hand over something like a national security letter and do whatever they want while making it clear to the heads of the company that anyone who talks or pushes back will rot in gitmo. Why wouldn't there be at least an equivalent to Room 641A (https://en.wikipedia.org/wiki/Room_641A) in every major US corporation that deals with massive amounts of people's sensitive data and communication?

Re:Reflections on Distrusting xz by acdha on Wednesday April 03, 2024 @07:28AM (#39916038)
> Most companies have more thorough processes to avoid this but that doesn't mean those processes are applied correctly everytime, especially if more than one malicious engineer is involved.

I’d also bet that you could exploit the tiers at many companies: how many places have more robust review for the staff engineers but then assume that some lowly “ops monkey” will take care of the build environment, etc.? I’d hope that wouldn’t work at Google, Microsoft, etc. but have heard enough stories about disparities between which jobs are contracted out and which have the coveted FAANG benefits that I wouldn’t exactly be shocked if it turned out otherwise.

Re:Reflections on Distrusting xz by thewanderer1983 on Wednesday April 03, 2024 @07:38AM (#39916120)
Deleted.
Re:Reflections on Distrusting xz by prmoustache on Wednesday April 03, 2024 @07:59AM (#39916309)
The thing is they only need one member sometimes, to observe what is in use.

Example scenario: "malicious engineer in say Microsoft, finds out that office365 is using xz internally and the library is pulled directly without code review. Same engineer or another member of same group would be that Jia Tan doing the necessary backdooring in xz to target office365. And bam all worlwide Office365 accounts would be backdoored."

I am not saying Office365 is using xz, I have no idea really, but this would be a possible scenario. I know MsTeams is using ffmpeg for example.

So I think having this discussion while only scoping linux distributions is a big mistake. xz project was particularly interesting as a target as it is distributed under BSD zero-close license, which is pretty much a public domain license. You don't have the attribution part of the BSD license so there are probably myriads of proprietary software using it too without them acknowledging it.

Re:Reflections on Distrusting xz by irdc on Wednesday April 03, 2024 @06:45AM (#39915715)
At this point, considering the apparent ease with which a project that is used pretty much everywhere was taken over, that seems like a reasonable position.
Re:Reflections on Distrusting xz by arp242 on Wednesday April 03, 2024 @06:58AM (#39915820)
I can walk out on the street and stab someone to death if I wanted to. This is surprisingly easy.

Just because something is relatively easy to pull off doesn't mean it happens a lot.

It's also not that easy to pull off because you need to have a project with relatively few eyes and a place to hide it. In this case: binary tests. But most projects don't have those.

There is no evidence for any of this, including that it's a nation-state actor. There's also a case to be made that it's NOT a nation-state actor as nation states use Linux and want a secure Linux. The NSA and such have somewhat conflicting interests here. We just don't know. It's likely we will never know.

All of this is starting to resemble the spy paranoia of the first world war. A few spies got caught and suddenly everyone was now a suspected German spy (including a general, if I recall correctly, who was detained for a while because he couldn't answer a question about baseball or some such).

I suspect that very soon people will start demanding maintainers put some of their blood in a Petri dish to be tested with a hot needle. Just in case.

Re:Reflections on Distrusting xz by mannykannot on Wednesday April 03, 2024 @07:51AM (#39916228)
Motive, opportunity, means - and consequences: it is primarily the absence of a motive, and secondarily the likelihood of consequences, that keeps the prevalence of street stabbings way lower that if means and opportunity were the only factors.

The argument against nation-states being involved has some problems: a state can avoid becoming victim to its own work, while its own restraint would not prevent developments elsewhere.

Re:Reflections on Distrusting xz by acdha on Wednesday April 03, 2024 @07:33AM (#39916076)
> There's also a case to be made that it's NOT a nation-state actor as nation states use Linux and want a secure Linux. The NSA and such have somewhat conflicting interests here. We just don't know.

I agree that we do not know that it’s a nation-state but this point seems to work in the opposite direction: this attack was very carefully constructed so only someone with a particular key pair could exploit it. That’s reminiscent of what the NSA did with the Dual EC constants, and they were confident enough about that to push it into the FIPS requirements for federal IT.

Re:Reflections on Distrusting xz by lolc on Wednesday April 03, 2024 @07:53AM (#39916254)
You're commenting under a link where commits to the xz-decoder are discussed. Some level of paranoia is warranted.

The binary files look like a sideshow in comparison. Maybe we're lucky the attacker was tempted to hide something in there.

Re:Reflections on Distrusting xz by Y_Y on Wednesday April 03, 2024 @06:58AM (#39915814)
I believe the term "nation state actor" is a term that means "country" but with the bonus of connoting that the writer is an armchair infosec wizard. This speculation is not valuable without adding information, otherwise it's just McCarthyist bluster.
Re:Reflections on Distrusting xz by MattPalmer1086 on Wednesday April 03, 2024 @07:32AM (#39916067)
Nation state actor is a standard info sec term. Using it does not imply any kind of wizardry.

Edit: most threat actors do not have the patience or the motive to behave in this way. It is reasonable to suppose that this is a nation state actor.

Re:Reflections on Distrusting xz by XorNot on Wednesday April 03, 2024 @08:02AM (#39916330)
There are organised crime networks which buy 0-days to run ransomware, and actively target companies to do it.

Why couldn't this be an attempt at finding or selling exploit access on the black market?

The problem here is no one is looking properly at the scope. It's more then trivial so everyone is leaping for "nation-state" as though that's the only threat actor with motivation and patience.

Re:Reflections on Distrusting xz by MattPalmer1086 on Wednesday April 03, 2024 @08:40PM (#39925182)
Granted, other threat actors remain a possibility, there is no proof.

It looks like the sort of thing nation states would do and develop. It could be some other group hoping to make money as you say.

Whoever they are, they seem to have good opsec, over multiple years.

Re:Reflections on Distrusting xz by HelloNurse on Wednesday April 03, 2024 @09:20AM (#39917099)
It's also a term that implies the competent, official hacking departments or espionage agencies of that country, rather than government-supported amateurs (e.g. an untrained policeman) or generic people from that country.
Re:Reflections on Distrusting xz by hyperhopper on Wednesday April 03, 2024 @07:30AM (#39916053)
Whales is a country but it is not a nation-state and probably does not have its own APT.
Re:Reflections on Distrusting xz by Y_Y on Wednesday April 03, 2024 @07:54AM (#39916258)
I'll bite. Wales is a country only in a "traditional" sense, since it does not currently hold the type of sovereignty we require of "countries" in the usual sense. As Voltaire said, "This body which called itself and which still calls itself the Holy Roman Empire was in no way holy, nor Roman, nor an empire.".

Wales, when independent, could reasonably be described as a nation state, being roughly associated with the Welsh people and their culture, language, history etc. But most countries are not nation states! The USA, Russia, and China for instance are explicitly plurinational.

If you think "nation" and "state" are synonyms (along with "country") then it's redundant to use both. If you think that "state" alone might lead someone to think of Alabama or Minas Gerais then say "nation". If you think "nation" will make people think of Cherokee, then say "country" or "sovereign state".

"Nation state" has a specific and well-established meaning, misusing it is like misusing any other jargon and just comes across as a failed attempt to seem part of the ingroup and hence authoritative.

(yes, apparently I will die on this hill)

Re:Reflections on Distrusting xz by stavros on Wednesday April 03, 2024 @08:40AM (#39916678)
Are there any actual nation states? Even small countries, like Greece, contain other nationalities.
Re:Reflections on Distrusting xz by Y_Y on Wednesday April 03, 2024 @09:02AM (#39916894)
Greece is certainly a nation state, being primarily occupied by "Greeks" (referring the the nebulous concept of a national identity). It doesn't matter if some French or Turks live there.

A country that strictly limited residency by ethnic identity might be called an "ethnostate" and indeed it's hard to find a pure example of one of those.

Re:Reflections on Distrusting xz by The_Colonel on Wednesday April 03, 2024 @07:30AM (#39916055)
> I believe this is a nation state actor

It is certainly possible, but we don't really have a good indication for that. This whole thing would be definitely doable by a single individual.

Re:Reflections on Distrusting xz by Barrin92 on Wednesday April 03, 2024 @08:00AM (#39916313)
doable yes, but what seems to me like a strong indication is the duration, multiple years, and the effort to set up a quasi patch infrastructure for the backdoor which I can't remember ever having seen in some amateur or ransomware hack.
Re:Reflections on Distrusting xz by epr on Wednesday April 03, 2024 @07:49AM (#39916209)
I've never been concerned about spies infiltrating open source projects compared to legitimate maintainers being hacked, even now after this whole xz incident.

I'll put it this way. Let's say a bad guy had a decent budget to spend on paying agents/criminals to break into maintainer's homes on their behalf with a rubber ducky, etc. I'd expect a pretty high success rate compromising their hardware...

Re:Reflections on Distrusting xz by dartos on Wednesday April 03, 2024 @08:21AM (#39916507)
You’re ignoring scale.

A single Jia Tian can be infiltrating 10s or more OSS projects each week without needing to travel around the world physically stealing hardware from various maintainers who they then need to impersonate.

They can just impersonate some anons with no real lives or connections and just get the keys to OSS projects given time.

Re:Reflections on Distrusting xz by epr on Wednesday April 03, 2024 @09:25AM (#39917154)
> A single Jia Tian can be infiltrating 10s or more OSS projects each week

Single? 10s or more per week?! I can't help but think you are underestimating the cost of developer time. How many hours of work did it take JT to infiltrate to the point of finally implementing a backdoor? How much does that time cost?

> just get the keys to OSS projects given time.

This is not what JT did though, and for good reason. Trust of anons in open source is generally built through contributions of real developer work over time. That does not scale.

> without needing to travel around the world physically stealing hardware from various maintainers

I wasn't suggesting stealing hardware to impersonate someone. I'm talking about hiring petty criminals or using field agents to break into a house, using physical hardware access to install a backdoor, etc. into the legit maintainers hardware. The field guy's goal is to not get caught, so the maintainer is unaware they are compromised.

I suppose the limitation with both approaches (maintainer plant vs compromising maintainers) is cost. My educated guess is that the cost of hiring skilled developers from a very limited pool for multiple years is more than it would cost to hire criminals that are already breaking into houses for low risk jobs where they don't even need to steal anything.

Re:Reflections on Distrusting xz by mrkramer on Wednesday April 03, 2024 @07:46AM (#39916184)
My assumption is that this was state sponsored mass surveillance campaign of some kind but God knows what exactly they were looking for.

I think if backdoor was discovered 2 or 3 months later, we maybe could understand better what they wanted to do. My speculation is that they wanted to build a massive botnet and then snoop on machines' processes and traffic looking for something. It's hard to speculate because luckily they were captured soon enough.

Re:Reflections on Distrusting xz by swed420 on Wednesday April 03, 2024 @09:52AM (#39917487)
I find it intriguing that out of all the speculative comment threads I've read so far, none of them have suggested it was Microsoft attempting to make FOSS look bad/vulnerable.
Re:Reflections on Distrusting xz by beardedwizard on Wednesday April 03, 2024 @10:37AM (#39918099)
How would that benefit Microsoft, who owns GitHub, the home of OSS? It's not a secret that oss is vulnerable, the opportunity for MS is to sell the solution to a captive audience.
Re:Reflections on Distrusting xz by swed420 on Wednesday April 03, 2024 @11:02AM (#39918407)
Microsoft making the decision to own GitHub in the first place also speaks to my suspicion. Embrace, Extend, Extinguish.
Re:Reflections on Distrusting xz by echelon on Wednesday April 03, 2024 @06:33AM (#39915626)
We could all be Jia Tan.

Someone could be bought, killed and replaced, or simply shadowed when they die or go to jail. Anonymity makes this even easier.

Re:Reflections on Distrusting xz by craftkiller on Wednesday April 03, 2024 @07:04AM (#39915872)
> killed [...] die or go to jail

All of my commits are signed with a PGP key that is on hardware security tokens and password-protected. In the event of my death, my digital identity could not be stolen without backdoors in my hardware security tokens.

That being said, $5 wrenches and large sums of money are still possible attack vectors.

Re:Reflections on Distrusting xz by acdha on Wednesday April 03, 2024 @07:43AM (#39916164)
Also don’t forget that not everyone expects perfection and a canny attacker can exploit that. It’s really easy to focus on how you’d avoid trojans, keyloggers, etc. but I’d also ask how likely it is that if someone sent a message from your email address claiming you’d lost your token in a minor accident, etc. that they’d believe it - or simply accept it if commits started showing up with a new key (maybe with an upgraded crypto system) since 99% of Git users never check those.
Re:Reflections on Distrusting xz by ccccccc1 on Wednesday April 03, 2024 @07:30AM (#39916051)
A cool tax-free no questions 500k can convince a lot of people
Re:Reflections on Distrusting xz by TheCondor on Wednesday April 03, 2024 @08:22AM (#39916516)
One thing I’ve learned, not from direct experience but from observation. These things are way cheaper than the more ethical and optimistic of us in society think. Your point is totally valid but the number is probably more like $5k-10k.
Re:Reflections on Distrusting xz by saagarjha on Wednesday April 03, 2024 @09:32AM (#39917229)
Tax free $500k? I don't want the IRS to come after me. Please mark all your bribes as regular income thanks
Re:Reflections on Distrusting xz by brabel on Wednesday April 03, 2024 @06:43AM (#39915695)
Everyone working on important open source code should have a real identity associated with them. The fact that "Jia Tan" was able to become a maintainer without anyone ever trying to figure out their real identity shows a huge weakness in our trust model in OSS (everyone real would have something like a Linked In page, Facebook, Twitter, Instagram, or better, their own website with stuff that could be used to ensure they're a real person - that could be faked as well but the amount of effort would be high, and checking this would be much better than just allowing effectively anonymous users to be maintainers - there's just no need for anonymity in this scenario!).
Re:Reflections on Distrusting xz by ninkendo on Wednesday April 03, 2024 @06:45AM (#39915716)
> everyone real would have something like a Linked In page, Facebook, Twitter, Instagram, or better, their own website with stuff that could be used to ensure they're a real person

Oof, I guess I’m not real then, as I have none of those things.

Re:Reflections on Distrusting xz by SanitaryThinkin on Wednesday April 03, 2024 @07:00AM (#39915839)
On top of what you mentioned I also dislike the TSA-like response the OSS community is taking with this happen stance.

I have anonomously contributed to many projects because I enjoy my privacy. All of my founding projects have also been done with anonymity.

Because someone wants their anonmity and privacy does not mean they're nefarious, and I find it funny the group that takes to these principals most is negging on those ideas.

Re:Reflections on Distrusting xz by xign on Wednesday April 03, 2024 @09:47PM (#39925672)
Personally, I do find it hard to trust an open source project maintained by an anonymous person. (I'm talking about maintainership, not regular contributions that need to be code reviewed by another maintainer) I may toy around with them but I will probably not use them in a manner where I need to trust them continuously.

It's totally cool for you to do whatever you want, since it's a free world after all, but if you want other people to use your code, then it's a two-way street no? Your code has a direct effect on their computers, and so they are placing their trust on you. You may value your privacy, but you need to balance that with other people's valuing their own security, and it's likely that whatever project you maintain may have an alternative as well.

If you just want to commit some code and not have people use them then that's another issue altogether.

I guess what I'm saying is: it's a two-way street. You can do things anonymously, but big companies / projects also don't have an obligation to use your code.

Re:Reflections on Distrusting xz by SanitaryThinkin on Thursday April 04, 2024 @07:43PM (#39937196)
> You can do things anonymously, but big companies / projects also don't have an obligation to use your code.

You're not wrong here, but I'm not forcing anyone to use my code bases or contributions.

Also, think about how many systems you blindly trust on a daily basis.

When you drive over a bridge, did you research the maitenance procedures and compliance was up to date?

When you got a house or apartment, did you look into the engineering sign offs and construction companies? And that maitenance has been done up to snuff? Even down to hoping the inspector knows what they're doing?

When you step into an elevator do you check the recent inspection plaque?

When you get on an airplane are you aware of its maitenance history? And to further my point by refering back to the house example, did the company even QA the plane before they shipped it?

And most importantly, did you check into whether the people actually did these things versus just saying they did them?

What kind of trust does having a persons name attached to the project actually provide? I would argue its a psuedo-facade trust basis that gives a false sense of security.

The truth of the matter is that you blindly trust millions of things on a daily basis. Including the very system you type from, which I guaruntee you has more than one anonymous maitainer attached to its underlying software.

I totally get where you're coming from, but the same problems exist in every industry, supply, politics, every facet of your life is based on many blind trust principles.

The one difference here with anonymous open source contributors is that they give you the code to read through yourself (and hope that you help ;) )

Very much unlike the proprietary software you're running beside it.

Re:Reflections on Distrusting xz by nottorp on Wednesday April 03, 2024 @07:30AM (#39916049)
I have Facebook and the account and what's on it is no one's -ing business in a professional context.
Re:Reflections on Distrusting xz by ghaff on Wednesday April 03, 2024 @07:51AM (#39916236)
I wouldn't post anything on Facebook (or on social media generally) that could be professionally embarrassing but I also don't generally accept invites from people who are solely professional acquaintances or use it in a purely professional context at all.
Re:Reflections on Distrusting xz by nottorp on Wednesday April 03, 2024 @07:59AM (#39916303)
> I wouldn't post anything on Facebook (or on social media generally) that could be professionally embarrassing

The age of self censorship :)

I don't post anything on my FB. I'd still reject any employer who wanted to take a look.

Re:Reflections on Distrusting xz by ghaff on Wednesday April 03, 2024 @08:05AM (#39916350)
Self-censorship is probably a good thing in many cases. And there are certainly things I don't care to share in writing on any public or semi-public medium.

But I agree that even if I can't keep an employer from sleuthing generally, I don't consider Facebook part of my professional record even if there's nothing on there I'd have a problem with a co-worker or potential co-worker seeing.

Re:Reflections on Distrusting xz by executesorder66 on Wednesday April 03, 2024 @08:15AM (#39916454)
>> what's on it is no one's -ing business in a professional context.

> The age of self censorship :)

I find this amusing.

I would also like to know what you hoped to achieve by self censoring the word fucking in your message above.

- HN doesn't block posts with any kind of "profanity" filter - You didn't spare anyone from the profanity, since we all knew exactly what you were saying/thinking

So I'm really curious what that actually achieved.

Re:Reflections on Distrusting xz by nottorp on Wednesday April 03, 2024 @09:01AM (#39916884)
That one amuses me because of a character in a (iirc) fantasy book that swore all the time but used just -ing everywhere. Sadly I don’t remember what character of which book…
Re:Reflections on Distrusting xz by jcurtis on Wednesday April 03, 2024 @09:46AM (#39917413)
It's Mr Tulip, of Terry Pratchett's The Truth.
Re:Reflections on Distrusting xz by nottorp on Wednesday April 03, 2024 @10:58AM (#39918349)
Thank you! I was pretty sure it was Pratchett (but not which book and character) but I self censored in case i was wrong :)
Re:Reflections on Distrusting xz by ed_elliott_asc on Wednesday April 03, 2024 @07:03AM (#39915860)
Also this can all be faked
Re:Reflections on Distrusting xz by reisse on Wednesday April 03, 2024 @06:51AM (#39915757)
No-one working on open source code on their time owes anything to anyone using the code. If you want an important open source project to be maintaned by non-anonymous person, surprise-surprise, hire that person and pay them.

Besides, some of the best open source contributors I know are almost-anonymous people behind nicknames and anime girls avatars.

Re:Reflections on Distrusting xz by brabel on Wednesday April 03, 2024 @04:30PM (#39922678)
It is not about owing someone... it's about having provenance of code.

If you're just an anonymous guy doing stuff for free and want to remain anonymous, that's fine, but then your software shouldn't be used by anyone who cares about toolchain attacks as there's just no way to trust you, and no way to verify every single commit you make on new releases.

For software that gets used by many, which is a goal of OSS (otherwise just don't even bother to publish stuff, what's the point?), there needs to be a face behind it.

I do agree with others that identity is a hard problem, but people here are pretending there's no solution to that (or misinterpreting what I wrote to mean people should have a Facebook or Twitter account, which is absolutely not what I was trying to say - I just mentioned the most popular websites real people are likely to be found on, as that could be used to prove their identity... for example, I have a Keybase account where my proof of identity, which is tied to my public keys, can be found on my GitHub profile - but they let you choose Facebook or Twitter for that purpose as well) when obviously there is. I should know, I work on this space.

Re:Reflections on Distrusting xz by reisse on Thursday April 04, 2024 @07:46AM (#39929105)
> If you're just an anonymous guy doing stuff for free and want to remain anonymous, that's fine, but then your software shouldn't be used by anyone who cares about toolchain attacks as there's just no way to trust you, and no way to verify every single commit you make on new releases.

What is, from security point of view, the difference between a toolchain attack performed by an anonymous contributor and by an identifiable real person?

> For software that gets used by many, which is a goal of OSS

It's not. The goal of OSS is to give users the possibility to study, change and improve the software. And that includes giving you ability to independently audit the code. All of that does not need any person behind it.

Re:Reflections on Distrusting xz by xign on Wednesday April 03, 2024 @09:53PM (#39925708)
At the same time no one is obligated to use your source code. I think the point here is from now on people (companies and large projects) may be more paranoid about anonymous contributors and refuse to sign off on using code maintained exclusively by them. It's fine for people to stay anonymous, but they just run the risk of not having the credibility for adoption and need to accept that.

But yes, I do trust certain figures like that, e.g. Asahi Lina. It's a fine ambiguous line. But at least in Asahi Linux there are real known human figures and they know who Asahi Lina is.

Re:Reflections on Distrusting xz by acdha on Wednesday April 03, 2024 @07:59AM (#39916307)
> everyone real would have something like a Linked In page, Facebook, Twitter, Instagram, or better, their own website with stuff that could be used to ensure they're a real person

Have you seen the campaigns people have run building fake LinkedIn profiles and slowly adding “connections”? There was one a few years ago which roped in a lot infosec people who should have known better and it’s gotten much worse with AI generators. Even before LLMs what you described would have been a godsend for intelligence agencies - who has more time for it, an open source developer writing actual code or the dedicated social media team at the IRA? – and now that’s increasingly worse.

Re:Reflections on Distrusting xz by brabel on Wednesday April 03, 2024 @04:40PM (#39922803)
I believe the solution to identity on the Internet needs to be tied to governments, that's unfortunate but in real life, that's always been the case and I can see no alternative here. Blockchain is a pipedream and no serious work is going to associate a person's identity to a key which cannot be revoked, cannot be recovered in case of "loss", can be tracked on a public ledger etc. etc...

But there's actual good work going on in the identity industry, like Verifiable Credentials, so this will become a reality soon: you will be able to verify someone's identity as long as you trust the issuer of their "credential" (which in the case here would mean basically a username and a public key or reference to a JWKS which can be used to verify the signature of the person, very much like the digital version of an identity card which can be used to check the signature on some piece of paper, but actually cyptographically safe)... so you would need to add a few governments to your list of "approved issuers", or something more indirect like universities (which themselves would rely on the government-issued identity) or traffic authorities (if you rely on driving licenses). Sure, Governments can lie, and people go to great lengths to steal others' identities in real life, but in the current world, we're still able to get bank accounts, passports etc. based on this model... just because the system is not perfect doesnt' mean it's not good enough, specially when there's no better alternative at all.

Re:Reflections on Distrusting xz by iron-s on Wednesday April 03, 2024 @06:48AM (#39915744)
What is real identity? Anything online can be faked. A state-issued id? How that protects against nation state?
Re:Reflections on Distrusting xz by brabel on Wednesday April 03, 2024 @04:45PM (#39922866)
Nothing protects you if you're up against a state. That doesn't mean we should give up completely.

Do you have a passport? That's a real identity in most places. Soon, it may be possible to use that to link your identity to a set of public keys which you can then use to identify yourself.

There's a lot of work to be done to make this a reality, but work is surely going on right now and this is going to be possible one day.

Check this out, as a starting point: https://curity.io/resources/learn/verifiable-credentials/

Re:Reflections on Distrusting xz by MattPalmer1086 on Wednesday April 03, 2024 @07:38AM (#39916123)
None of those things prove identity. A well funded or just patient attacker can spoof all of those. Sure, it raises the bar a tiny bit, but it's no proof of identity.
Re:Reflections on Distrusting xz by nottorp on Wednesday April 03, 2024 @07:27AM (#39916027)
Why would I trust an "important open source project" with my identity?

It goes both ways.

Besides, the 'state actor' the security theater people keep mentioning would have no trouble creating such real identities.

Re:Reflections on Distrusting xz by brabel on Wednesday April 03, 2024 @04:49PM (#39922925)
If you don't trust the project, you wouldn't contribute to it.

The state actor may be able to fake identities, but that would still allow tracking the identity to a particular state... and if caught multiple times, that state would start losing credibility and projects may choose to stop trusting people from such nationality, unfortunately, or at least require more strong evidence the person is real and trustworthy if they come from known rogue nations.

Re:Reflections on Distrusting xz by nottorp on Wednesday April 03, 2024 @06:20PM (#39924005)
> If you don't trust the project, you wouldn't contribute to it.

Trust them to merge a bugfix is different from trusting them with my identity isn't it?

There are degrees of trust. For example I have a gmail address in my profile because the spam filter on there is better than what I have on my personal domain. People I've known for longer, business or otherwise, get the other (that I read more often).

Re:Reflections on Distrusting xz by executesorder66 on Wednesday April 03, 2024 @07:00AM (#39915832)
News just in: NSA et al. defeated after having to create a LinkedIn and Instagram profile for their agents.
Re:Reflections on Distrusting xz by j-krieger on Wednesday April 03, 2024 @06:58AM (#39915812)
Who are you to propose requirements onto people who work for free?
Re:Reflections on Distrusting xz by brabel on Wednesday April 03, 2024 @04:53PM (#39922967)
I am not imposing anything on anyone. I am only saying that an OSS project that aims to be used as part of important infrastructure should impose at least some sort of identity vetoing and not just make random anonymous users maintainers of anything.

If your project is not important and you don't care about any of this security stuff, feel free to continue publishing your untrustable projects.

Re:Reflections on Distrusting xz by tomn on Wednesday April 03, 2024 @07:39AM (#39916137)
Yeah, it's not enforced (and certainly not with linked-in and facebook) but it's really not uncommon to require use of real names for contributions.

Linux doesn't allow anonymous contributions:

https://www.kernel.org/doc/html/latest/process/submitting-pa...

and this guide has been adopted by a lot of GPL-licensed projects (at least openwrt, glibc and gcc).

Re:Reflections on Distrusting xz by ghaff on Wednesday April 03, 2024 @07:58AM (#39916301)
I'm not sure why the downvotes. That seems to be a statement of fact.

You can do a certain amount of identity obfuscation online but for anyone with a real professional profile you're generally not really anonymous if anyone really cares to find out your true name.

Re:Reflections on Distrusting xz by tomn on Wednesday April 03, 2024 @08:58AM (#39916844)
Me neither, i even provided a source, and it's easy to find other examples. There certainly are projects that allow anonymous contributions, but i doubt it's the majority of projects that one would consider important.

For these kinds of projects you could make up an identity relatively easily and nobody would know, but you're screwing over the project (as they may need to remove your contributions if they find out), so it's not something to be doing if you actually want to contribute (instead of inserting backdoors).

The original idea (not being able to contribute without a verified identity) is still wrong, but it's wrong because it's impractical to prove identity in a way that people find acceptable (and works), not because people will not give up anonymity, as many of the replies state.

Re:Reflections on Distrusting xz by ghaff on Wednesday April 03, 2024 @09:19AM (#39917088)
There are people who downvote things that present facts that aren't in accordance with how they think the world should be.

I do think it's difficult to verify identity in any reasonably acceptable lightweight way. That said, for the larger projects I'm most familiar with, a lot of people work for companies, attend conferences, etc. They may go by nicknames day to day, but they have known real identities and their professional existence wouldn't be possible without one.

Re:Reflections on Distrusting xz by xign on Wednesday April 03, 2024 @09:58PM (#39925730)
Wasn't there some controversies around this before? I remember there was some talk of why Asahi Lina (anonymous vtuber working on Asahi Linux) can contribute code to Linux. From casual search: https://www.spinics.net/lists/kernel/msg4888830.html

FWIW I like Asahi Lina, just trying to understand the discrepencies

Re:Reflections on Distrusting xz by tomn on Friday April 05, 2024 @12:11PM (#39944140)
Interesting. My understanding is that these projects don't allow anonymous contributions to make their copyright situation clear, so in theory if marcan42 sent a letter to the linux project saying that contributions from Asahi Lina are actually theirs, they might reasonably be fine with that.

It seems like this is how the ASF runs: you can be anonymous publicly, but you have to sign their CLA (or whatever they call it) properly.

To me, the people trying to unmask Asahi Lina are being simultaneously mean and silly. If it's so obvious that it's marcan42 doing a voice, do you really need to point it out? That's kind of the joke.

Re:Reflections on Distrusting xz by the8472 on Wednesday April 03, 2024 @07:05AM (#39915881)
If you want security, pay for independent code audits (not compliance bullshit). Repeatedly. Don't offload your desires onto one-man-shows that the world decided are useful tools.
Re:Reflections on Distrusting xz by berniedurfee on Thursday April 04, 2024 @07:47AM (#39929111)
When you find one cockroach, you can be sure there are thousands more you haven’t found.