News for hackers. Stuff that matters.
Re:Debian on xz-utils: revert to version that does not conta... by toxik on Saturday March 30, 2024 @06:33AM (#39873527)
When is Google, Amazon, Facebook, the DARPA, the EU - ANYONE going to step up to the plate and fund developer time for such obviously critical infrastructure? Why could a single Chinese guy undermine the entire system with so little effort?
Re:Debian on xz-utils: revert to version that does not conta... by nindalf on Saturday March 30, 2024 @07:21AM (#39873789)
Everything looks obvious in hindsight doesn’t it? Several of the organisations you’ve mentioned have funded projects to rewrite critical internet infrastructure in a memory safe language - HTTP client, HTTP reverse proxy, DNS resolver, NTP server, TLS library, AV1 decoder, a replacement for sudo, among other projects (https://memorysafety.org). More can always be done, and it can always be done faster than it’s being done.

Fwiw, it’s possible to use zstd instead of xz, which was created at and actively maintained at Facebook.

So if you were expecting Big Tech to identify this class of problem and do something about it, it seems like they are. But if you expected them to eliminate every single possible instance of such an issue, that’s a bit much isn’t it?

Re:Debian on xz-utils: revert to version that does not conta... by DrBazza on Saturday March 30, 2024 @07:27AM (#39873819)
Would a 'memory safe' language really have stopped any of this? From I've briefly read this incident the whole attack process (PR, fake test data, social engineering) from start to finish by a state-actor (most likely), could have been in any language.
Re:Debian on xz-utils: revert to version that does not conta... by bostik on Saturday March 30, 2024 @08:13AM (#39874101)
> could have been in any language

Bingo. This was a highly motivated, targeted, and well planned compromise of the supply chain itself. And funnily enough, using a memory-safe language could have actually made it less likely for this to be detected.

There were symptoms showing up prior to the discovery, most prominently Valgrind errors in otherwise unrelated project builds. (ref: https://bugzilla.redhat.com/show_bug.cgi?id=2267598)

Re:Debian on xz-utils: revert to version that does not conta... by dwattttt on Saturday March 30, 2024 @08:52AM (#39874377)
The equivalents in memory safe languages raise more eyebrows. Segfault developing on a shared C codebase and it's a day ending with y. Segfault in a language that's meant to not let that happen? Or conversely if a patch turns up that's explicitly doing lots of unsafe things? That gets more scrutiny.
Re:Debian on xz-utils: revert to version that does not conta... by bostik on Saturday March 30, 2024 @09:49AM (#39874864)
People like to harp on software written in C, and often for very good reasons. A vocal minority of those then whip out their calling card, "should have used Rust!"

As a result, there is a sentiment in the peanut gallery that "memory safe language == Rust".

You are correct in that a segfault in a project written in memory safe language gets more scrutiny. But imagine for a moment what would have happened if the compiler guaranteed that those Valgrind-caught memory access violations could not have occurred in the first place?

We got lucky, and the developers who wrote the actual backdoor were sloppy. They were aiming for sneakiness and evasion first, memory correctness would have likely been an afterthought. The next group tasked with a similar approach will know better.

Re:Debian on xz-utils: revert to version that does not conta... by lifthrasiir on Saturday March 30, 2024 @07:46AM (#39873936)
You are right, but C/C++ was vulnerable to this particular MO because of its complicated build process.
Re:Debian on xz-utils: revert to version that does not conta... by kukkamario on Saturday March 30, 2024 @08:15AM (#39874114)
There are million ways to hide exploits even in "safe" languages. If you have a bad actor that can spend couple years prepping how to hide an exploit, then it doesn't really matter which programming language is used.

Build system is complex because of legacy and need to support millions of different platforms and distros.

Re:Debian on xz-utils: revert to version that does not conta... by lifthrasiir on Saturday March 30, 2024 @08:21AM (#39874149)
Different platforms and distros still share a handful of common environments, while autoconf was invented back when there were hundreds of different environments to care about. Xz doesn't depend that much on the platform and could have ditched autoconf at least in the theory. But such switch is annoying in the world of C/C++.
Re:Debian on xz-utils: revert to version that does not conta... by josephg on Saturday March 30, 2024 @08:41AM (#39874300)
I'd say there are fewer ways to hide exploits like this. Rust doesn't use configure scripts. And it has a built in testing mechanism - so there's no need for makefiles to run scripts. Any build scripts as complicated as configure in a rust project would be a much bigger red flag. And it would be rare and weird to see GNU_IFUNC in use in a rust project too.

Sure; you can hide nasty code in rust. But I think it would have been harder. Not because rust is memory safe. But because rust is decades newer, and it doesn't inherit all of the quirks in C, in its build system and obscure mechanisms for dynamic linking.

Re:Debian on xz-utils: revert to version that does not conta... by uecker on Saturday March 30, 2024 @08:03AM (#39874050)
I would say today there is no real need for autoconf anymore in C / C++ and the build process could be fairly easy.
Re:Debian on xz-utils: revert to version that does not conta... by lewis1028282 on Saturday March 30, 2024 @09:20AM (#39874603)
It's easy to hide things in the Python build process as well. Things like using `getattr` instead of . (dot) etc
Re:Debian on xz-utils: revert to version that does not conta... by saati on Saturday March 30, 2024 @08:20AM (#39874148)
Yes, they used a memory corruption bug to run the payload.
Re:Debian on xz-utils: revert to version that does not conta... by DrBazza on Saturday March 30, 2024 @08:26AM (#39874176)
And if the code submitted used Rust's 'unsafe'?
Re:Debian on xz-utils: revert to version that does not conta... by dralley on Saturday March 30, 2024 @09:33AM (#39874707)
Then it would be a massive shining "review me harder daddy" beacon
Re:Debian on xz-utils: revert to version that does not conta... by armitron on Saturday March 30, 2024 @08:32AM (#39874215)
You're imagining things, there is no memory corruption mechanism. The payload is delivered into the process address space by dynamic linking through a shared dependency and hijacks a function via GNU_IFUNC.
Re:Debian on xz-utils: revert to version that does not conta... by nindalf on Saturday March 30, 2024 @08:37AM (#39874258)
Well, Rust doesn’t support dynamic linking. So this particular attack wouldn’t be possible with Rust in March 2024.

But it’s a moot point because there are undoubtedly other mechanisms a malicious maintainer could use to inject a vulnerability. Just not this exact one.

Re:Debian on xz-utils: revert to version that does not conta... by josephg on Saturday March 30, 2024 @08:47AM (#39874344)
Yes, But it would at least be harder. C has so many weird quirks and edge cases - including decades of legacy mechanisms (like IFUNC) that almost nobody today understands.

The underhanded C contest would be a lot less entertaining in rust. Thats a good thing.

It wouldn't solve the underlying problem permanently. But I don't think there's any slam-dunk solution here. I'll take any small wins where we can get them.

Re:Debian on xz-utils: revert to version that does not conta... by steveklabnik on Saturday March 30, 2024 @12:21PM (#39876251)
There was an Underhanded Rust contest many years ago https://news.ycombinator.com/item?id=15352253
Re:Debian on xz-utils: revert to version that does not conta... by dralley on Saturday March 30, 2024 @09:37AM (#39874747)
Rust does support dynamic linking, it's just not that useful because of the unstable ABI. You can dynamically link against C code, or Rust code presenting a C ABI, without issue, and you can dynamically link against Rust code if you accept the inherent limitations at the moment
Re:Debian on xz-utils: revert to version that does not conta... by jppittma on Saturday March 30, 2024 @08:59AM (#39874441)
As far as I can tell, Rust _does_ support dynamic linking. You can both compile rust code to .so and link against existing shared objects. What are you talking about?

>The compiler supports various methods to link crates together both statically and dynamically. This section will explore the various methods to link crates together, and more information about native libraries can be found in the FFI section of the book.

https://doc.rust-lang.org/reference/linkage.html

Re:Debian on xz-utils: revert to version that does not conta... by nindalf on Saturday March 30, 2024 @09:51AM (#39874881)
> What are you talking about?

I’m talking about what happens in practice, which is shipping static binaries that contain all of their dependencies, except for libc.

Re:Debian on xz-utils: revert to version that does not conta... by ReflectedImage on Saturday March 30, 2024 @08:43AM (#39874325)
Well Valgrind originally went off on it and oss-fuzz presumably would have caught it since here's a PR from the attacker asking for it to be disabled: https://github.com/google/oss-fuzz/pull/10667
Re:Debian on xz-utils: revert to version that does not conta... by moonchild on Saturday March 30, 2024 @08:34AM (#39874235)
capability safety inhibits supply chain attacks. memory safety is a prerequisite, but it is not sufficient
Re:Debian on xz-utils: revert to version that does not conta... by nindalf on Saturday March 30, 2024 @09:18AM (#39874588)
Right now they’re worried that somewhere in those 700 commits added by the malicious maintainer there may be some subtle memory safety bug. So great is the difficulty of finding these bugs that they’re planning to discard all work since the commiter became involved, including work by others.

Introducing such hidden bugs that are basically impossible to find is much harder in a memory safe, statically linked language.

Re:Debian on xz-utils: revert to version that does not conta... by Anon1096 on Saturday March 30, 2024 @06:46AM (#39873591)
There was just 2 active maintainers on xz, and the compromised individual has been the primary for >1 year. If anyone were to fund development time on it, ironically it would go exactly to the person working to backdoor it.
Re:Debian on xz-utils: revert to version that does not conta... by toxik on Saturday March 30, 2024 @06:58AM (#39873659)
Facebook would fund a Facebook engineer and would therefore have vetted the person in question. Same for other actors, there are ways to support these projects with eg a non-profit organization bankrolled by EU taxes or similar. Imagine if instead of Some Guy being the sole maintainer of critical infrastructure piece X, there was shared responsibility and accountability and less personal attachment.

We can argue about the specific implementation, but to suggest that it’s impossible seems incorrect to me.

Re:Debian on xz-utils: revert to version that does not conta... by buffet_overflow on Saturday March 30, 2024 @07:16AM (#39873764)
> Facebook engineer and therefore have vetted

Haven’t nation state actors openly infiltrated high level companies? This would provide a false sense of security imo. If anything, we need better testing and behavior heuristics for incoming code.

Re:Debian on xz-utils: revert to version that does not conta... by bbarnett on Saturday March 30, 2024 @08:04AM (#39874053)
One thing missing here.

I develop cool thing X. Now, 100 minor things depend upon it.

Suddenly, Facebook (or anyone of that size!) starts using it, and decides to vet the maintainer/author.

Who says anyone has to cooperate? It's his software. He wrote it. Don't like it?

Well tough!

Now obviously Facebook could author a replacement. It could fork and maintain.

But the very nerve that Facebook(or anyone!) would insist upon a security audit of the anonymous author would be very, very strange.

Next up, I lend a neighbour my lawn mower, after he comes begging to borrow it. Oh but wait! My neighbour now wants me to sign a libabilty form, and also undergo a security check, all so he can borrow my lawnmower!

The hell?!?!

Hoping this illustrates my point. The project author owes nothing to anyone.

And it gets more wacky, if there are 100 companies demanding audits. What? Demand?!

This is where distros are the strong point. They aren't perfect, but they catch a lot of stuff on their own. And maintainers of different distros often backchannel, support each other in this.

In terms of some government org "vetting" people? Way to take the last vestiges of free software, and hacking, and turn it into a gatekeeping, bureaucratic nightmare. I guess one will need credentials, government id, a 10 year security check, to be fingerprinted, and so on? Security clearances work like that, and that's how you vet someone.

Re:Debian on xz-utils: revert to version that does not conta... by buffet_overflow on Saturday March 30, 2024 @08:35AM (#39874237)
Agreed that the vetting of private people would be invasive and not a good use of resources. It would also not work for recently compromised accounts.

I’m also personally torn on how much we want giant private companies controlling more and more of the core compute infrastructure and software.

In an ideal world, the code and software itself would be automatically analyzed for malicious use cases in release and deployment pipelines, but that’s a magic hand wavy kind of ask of a huge magnitude and complexity.

Re:Debian on xz-utils: revert to version that does not conta... by darnir on Saturday March 30, 2024 @08:39AM (#39874287)
This exact thing happened to me. I maintain a fairly popular free software project. A few years ago, I received an email from a nasa.gov domain claiming that they want to use the project internally and are auditing all their suppliers. They wanted documentation on me and on how I audit my supply chain for the project. Not cool. I don't have time for these shenanigans in my personal time.
Re:Debian on xz-utils: revert to version that does not conta... by jdiez17 on Saturday March 30, 2024 @09:21AM (#39874610)
I'm sure you had the choice not to provide that documentation, right?
Re:Debian on xz-utils: revert to version that does not conta... by darnir on Saturday March 30, 2024 @10:52AM (#39875438)
Well, yes. And I chose to exercise that right.

The point of the anecdote is to supplement the parent poster by stating that their hypothetical scenarios are already happening.

We should have better testing and more eyes on incoming code for projects we depend on. But my point I guess is that vetting maintainers is not an option.

Re:Debian on xz-utils: revert to version that does not conta... by bdd8f1df777b on Saturday March 30, 2024 @09:31AM (#39874680)
And in this case Facebook did author a replacement--zstd. It just didn't get popular enough. And even when it does get popular, it won't replace all usages of xz, only some of them.
Re:Debian on xz-utils: revert to version that does not conta... by KeplerBoy on Saturday March 30, 2024 @09:04AM (#39874488)
The problem is that a sole maintainer was able to shove code all the way into distros. You can vet people all you want, it won't prevent these issues as long as we all run code, which has never been looked at by a second pair of eyes.

Same reason why commercial planes always have two vetted pilots in the cockpit and ATC watching their every move. People can and will do malicious shit, if their mental health declines.

Re:Debian on xz-utils: revert to version that does not conta... by ddalex on Saturday March 30, 2024 @06:35AM (#39873536)
How do you know it is a guy, and it is Chinese?
Re:Debian on xz-utils: revert to version that does not conta... by nindalf on Saturday March 30, 2024 @07:14AM (#39873752)
At best there’s an account with a Chinese sounding name. I’m never going to carry water for the PLA or the CCP, but I’d need to see stronger evidence before agreeing it’s them.
Re:Debian on xz-utils: revert to version that does not conta... by mathfailure on Saturday March 30, 2024 @08:01AM (#39874037)
What about water for ZND or CBSOM?
Re:Debian on xz-utils: revert to version that does not conta... by nindalf on Saturday March 30, 2024 @09:53AM (#39874898)
If you’re implying that I used incomprehensible initialisms, I’d argue that they’re easily understandable in the context of an alleged state sponsored attack from China.

I think your comment is facetious and adds nothing to the conversation.

Re:Debian on xz-utils: revert to version that does not conta... by mathfailure on Monday April 01, 2024 @01:26PM (#39896601)
I as well think your comment adds nothing to the conversation.

Neither this one, nor the previous one.

Re:Debian on xz-utils: revert to version that does not conta... by toxik on Saturday March 30, 2024 @07:01AM (#39873677)
I take it you haven’t looked into the details of this, there is no proof but there is strong evidence in name choice, language use and the MO.
Re:Debian on xz-utils: revert to version that does not conta... by bdhcuidbebe on Saturday March 30, 2024 @08:07AM (#39874075)
I strongly disagree. If you scratch the surface you find ”misoeater” and some russian sounding user names, and even a danish name. These are all fake profiles and gives us no information about the bad actor.

Spreading doubt is taught in bad actor-academy.

Re:Debian on xz-utils: revert to version that does not conta... by t-sauer on Saturday March 30, 2024 @07:44AM (#39873922)
That is rather naive thinking. Making up a misleading identity is probably one of the first things someone with such a long term plan would do.
Re:Debian on xz-utils: revert to version that does not conta... by jeltz on Saturday March 30, 2024 @07:14AM (#39873754)
Name choice I disagree with. The attacker used at least Chinese, Indian and Scandinavian names when working on this. It is possible we can find out something by doing a statistical analysis of the language but as far as I know no such analysis has been done.

Edit: A Russian sounding alias was also used.

Re:Debian on xz-utils: revert to version that does not conta... by kijin on Saturday March 30, 2024 @07:25AM (#39873807)
My current hypothesis is that the attacker(s) deliberately chose a Chinese-sounding name to make people jump to conclusions. A sophisticated state actor would be no stranger to making their actions look like somebody else's.

Or maybe they actually are Chinese, and want me to believe the sentence above. :)

Re:Debian on xz-utils: revert to version that does not conta... by lodovic on Saturday March 30, 2024 @08:57AM (#39874426)
Indeed - seeing the level of sophistication and long term thinking behind this attack, it seems everything about it was planned meticulously, including usernames, attack vectors, communication with the community, and when to commit the parts of the exploit. There's probably a full team of people behind it and attributing it to China because of the username may be too quick.
Re:Debian on xz-utils: revert to version that does not conta... by raverbashing on Saturday March 30, 2024 @08:00AM (#39874023)
And time zone as well it seems
Re:Debian on xz-utils: revert to version that does not conta... by cpach on Saturday March 30, 2024 @07:47AM (#39873937)
Good point. We just don’t know.
Re:Debian on xz-utils: revert to version that does not conta... by rurban on Saturday March 30, 2024 @07:56AM (#39873999)
He is Taiwanese. At least he posted on Taiwanese boards.

Jie would be female, Jia male

Re:Debian on xz-utils: revert to version that does not conta... by yorwba on Saturday March 30, 2024 @08:47AM (#39874346)
Occurrences of "jiat0218" on Taiwanese websites are clearly copypasta from some other website that's probably defunct now.

While gendered names do exist, you can't easily tell from the romanization, it depends on the exact characters used. And most likely "Jia" is the surname, or rather the romanization of one of five different surnames.

Re:Debian on xz-utils: revert to version that does not conta... by programmeruser on Saturday March 30, 2024 @05:52PM (#39879025)
Tan is a common Chinese surname that's the Hokkien/Teochew transliteration of Chen. It's more commonly seen in overseas Chinese but I wouldn't jump to any conclusions
Re:Debian on xz-utils: revert to version that does not conta... by mrintegrity on Saturday March 30, 2024 @03:10PM (#39877677)
In other discussions the Chinese person in question has been spoken about at length
Re:Debian on xz-utils: revert to version that does not conta... by twno1 on Monday April 01, 2024 @08:52PM (#39901193)
The commit history from bad actor set tomezone to +0800 Source: https://boehs.org/node/everything-i-know-about-the-xz-backdo...
Re:Debian on xz-utils: revert to version that does not conta... by lco on Tuesday April 02, 2024 @06:41AM (#39904228)
> It’s most likely our actor just mashed plausible-sounding Chinese names together.

Also from that writeup. It also links to https://rheaeve.substack.com/p/xz-backdoor-times-damned-time... which suggests the actual TZ of 'Jia Tan' to be UTC+02/03