News for hackers. Stuff that matters.
Re:Reflections on Distrusting xz by constantcrying on Wednesday April 03, 2024 @06:51AM (#39915758)
>Which raises the concerning question of how much more sleeper maintainers there are.

Given how easy the infiltration is and how extremely hard to detect it is, likely a lot.

For an intelligence operation it is also extremely cheap, you just need a few knowledgeable developers spending some time each week on the project. The upside being a backdoor into a significant portion of infrastructure, the downside being wasted time.

I do not think it is unlikely that in many important open source software projects there are one or two people assign to keep an eye on things. They don't even need to be malicious, just being somewhat trusted contributors is enough. I would be extremely surprised if the NSA hasn't a couple of guys who keep watch on the Linux Kernel.

Re:Reflections on Distrusting xz by moritonal on Wednesday April 03, 2024 @07:00AM (#39915834)
The irony is that this would make a oddly effective way of having paid open source devs who for the most part just honestly improve projects. With the massive downside they undermine it at a critical moment.
Re:Reflections on Distrusting xz by rightbyte on Wednesday April 03, 2024 @07:52AM (#39916241)
Ye then maybe thanks to them finally we'll have The year of the air gapped Linux desktop.

I dunno what to do if e.g. Debian gets compromized, as in, I can't trust the collective of maintainers.

I assume any Windows machine is backdoored. Trivially proven by forced auto updates.

Maybe air gapping some home computer for sensitive data might be a good idea.

Re:Reflections on Distrusting xz by pants2 on Friday April 05, 2024 @12:13AM (#39938614)
The ideal situation is having multiple intelligence agencies all working on one project and spotting each others' backdoors, so at the end of the day we just have a really secure and well-maintained project.
Re:Reflections on Distrusting xz by david_draco on Wednesday April 03, 2024 @08:46AM (#39916738)
> Given how easy the infiltration is and how extremely hard to detect it is, likely a lot.

I read it exactly the other way around: the infiltration took years and detecting it was the default with a fuzzer, which had to be disabled for the exploit to succeed.

It speaks to the hardening and that hardening should be required more. And of course the precarious roles of maintainers, which have been discussed elsewhere.

Re:Reflections on Distrusting xz by dullcrisp on Wednesday April 03, 2024 @11:03AM (#39918419)
The exploit could be detected with a fuzzer perhaps. The infiltration seems like it was something that would be very easy for a funded intelligence agency to do, and would be nigh undetectable if they had been subtly introducing bugs rather than shipping a sophisticated backdoor to every Linux distro.

You have to assume if this was an intelligence agency they didn’t burn their only agent like this.

Re:Reflections on Distrusting xz by HankB99 on Wednesday April 03, 2024 @02:20PM (#39921031)
> they didn’t burn their only agent like this.

I think I'd characterize that as "their only identity." What's the chance that some number (>1) of actual agents were sharing this identity? If that's the case, I'd extrapolate that to multiple identities. In fact the social engineering to gain the trust of the original maintainer likely involved several identities.

I suspect that detectability of intentionally injected bugs would be very low.

Re:Reflections on Distrusting xz by Thorrez on Wednesday April 03, 2024 @11:56PM (#39926374)
>a fuzzer, which had to be disabled for the exploit to succeed.

According to this comment, the fuzzer wouldn't have detected it. It wasn't necessary to disable the fuzzer:

>https://news.ycombinator.com/item?id=39911249

Re:Reflections on Distrusting xz by geggo98 on Thursday April 04, 2024 @12:38AM (#39926587)
You are right, it took quite some time. On the other hand, it looks like the legitimate part of contributing to xz was only a part time job for the attacker. The rest of the time, they either worked on the exploits, or in other things, like infiltrating other projects using a different handle.

Basically I can imagine the attackers being a well organized group, using work sharing and pipelining. Some members of the group would be preparing exploits, some would infiltrate projects and some would make sure not to get caught. And since infiltrating takes time, they would make sure to have multiple projects in the pipeline, seine in the early contributor stage, some in the social pressure stage, and some in the exploiting stage.

Re:Reflections on Distrusting xz by jannes on Wednesday April 03, 2024 @09:03AM (#39916898)
At this point I wouldn’t be surprised if the NSA also has a couple of Microsoft employees on their payroll.
Re:Reflections on Distrusting xz by cactusfrog on Wednesday April 03, 2024 @09:49AM (#39917445)
I believe LLMs could be useful here as a component of pre-commit hook.
Re:Reflections on Distrusting xz by Gormo on Wednesday April 03, 2024 @10:00AM (#39917614)
> Given how easy the infiltration is and how extremely hard to detect it is, likely a lot.

This particular case seems to be an example of the exact opposite. It took "Jia Tan" two years to conduct all of the social engineering necessary to get into a position to introduce the backdoor, then upon doing so, was caught almost immediately, with the initial discovery not even coming from a dedicated security researcher, but from a sysadmin who kept digging when he saw unusual performance issues.

And the threat actor here deliberately went after the weakest link in the chain. Assuming that the goal was to compromise sshd, the antagonist evidently found it too much of a challenge to attempt to infiltrate OpenSSH itself, so targeted a small-team compression project that only some deployments even link to, and still got caught extremely rapidly.

Re:Reflections on Distrusting xz by constantcrying on Wednesday April 03, 2024 @11:11AM (#39918515)
I don't think you are looking at it the right way. Two years matter if you are a hobbyist or your goal is to compromise the system for some individual gain.

For a nation state actor two years is nothing. Likely the entire attack didn't cost more than a couple of thousand hours of developer time. I would guess it was easily cheaper than 100k in financial terms, that is extremely efficient for an intelligence operation with the upside being access to a large amount of servers.

The method by which he got caught also depended largely upon random chance, some major "if's" needed to happen, the performance reduction was an unfortunate side effect from the perspective of the attacker, really a minor mistake exposed him. Even if he were exposed a couple months later the damage would have been enormous, if that version had become a stable part of any major distro for the next few years hundreds of thousands of machines would have been vulnerable.

There is absolutely no reason to assume that if another attack of this quality happens it will not find it's way into some stable distro.

Re:Reflections on Distrusting xz by Gormo on Wednesday April 03, 2024 @11:45AM (#39918970)
> For a nation state actor two years is nothing.

Perhaps it is, perhaps not. If they are playing a long game, two years might be a straightforward investment. But the point is that it still took two years either way -- the fact that something is only feasible for big institutional actors implies that it is not in itself "easy".

And the kinds of organizations that have the motivation and resources to engage in this kind of long-game infiltration by definition have the resources to influence, infiltrate, and manipulate in a variety of ways. It is absolutely not clear here whether having formal organizations involved in directing the development of xz would have made it easier or harder for whomever was behind this attack.

> The method by which he got caught also depended largely upon random chance,

Why do you presume it's random chance, and that it wouldn't be better represented by more complex probability models? On the surface, it seems like the chance of overall detection would be reasonably represented as an equivalent of an MTBF calculation, with the number of skilled users capable of detecting the vulnerability through normal use as an input. (Not to say that we have enough data to calculate this probability, or that this necessarily even happens enough to have a statistically significant sample -- it might all be "black swans" -- but there's certainly non-random stochastic inputs into this.)

> There is absolutely no reason to assume that if another attack of this quality happens it will not find it's way into some stable distro.

There's no guarantee for any of this. All rules can be gamed, all incentives can be manipulated, all organizations can be compromised. There is no perfect solution to security, and no guarantee that any prescriptive solutions based on generalizing from the specifics of one incident would not create more opportunities for attackers rather than fewer.

Re:Reflections on Distrusting xz by constantcrying on Wednesday April 03, 2024 @12:32PM (#39919525)
I don't disagree, I am not even sure what exactly you are arguing against.

I completely agree with your comment about the detection being the result of iterative low probability chances of detection. My point was that, even if that is the case, there was a major chance that it would have gone into many stable distos or would have been exploited.

Re:Reflections on Distrusting xz by Gormo on Wednesday April 03, 2024 @01:23PM (#39920302)
> I don't disagree, I am not even sure what exactly you are arguing against.

I am arguing that this incident demonstrated the resilience of the FOSS model to even extremely strategic, long-term compromise attempts. Some antagonist invests years worth of time, effort, and money to introduce this backdoor, and the openness of the entire process allowed a single engineer to unravel the whole thing almost immediately after the backdoor was introduced, and every distro immediately sprung into action and neutralized it entirely.

All of the people complaining about the vulnerability of small projects, the pseudonimity of contributors, the need for more institutional involvement, etc. are getting things exactly backwards.

Imagine if this were a closed-source project funded and managed by an opaque institution, and the attacker used a different set of social engineering tactics to backdoor the code from within the org. Suppose then that Andreas Freund noticed the exact same behavior and set out to investigate it. How far would he have gotten?

This incident validates the "many eyes" concept and is a point in favor of the FOSS model.

Re:Reflections on Distrusting xz by kevans91 on Wednesday April 03, 2024 @01:29PM (#39920385)
fwiw, characterizing Andres as a sysadmin isn't really the whole picture; he's a postgres developer that conducts benchmarking operations with some frequency (and he's quite good at what he does)... he's perhaps naturally a bit more sensitive to things like the cumulative effect of 500ms or so over a number of sshd invocations.
Re:Reflections on Distrusting xz by Gormo on Friday April 05, 2024 @11:30AM (#39943624)
You're right -- I went back and changed "sysadmin" to "engineer". Either way, though, he was not a dedicated security researcher, and managed to unravel this entire thing upon noticing an anomaly in the course of his regular work.