News for hackers. Stuff that matters.
Re:Backdoor in upstream xz/liblzma leading to SSH server com... by ldayley on Friday March 29, 2024 @04:12PM (#39868499)
If this question had a reliable (and public) answer then the world would be a very different place!

That said, this is an important question. We, particularly those us who work on critical infrastructure or software, should be asking ourselves this regularly to help prevent this type of thing.

Note that it's also easy (and similarly catastrophic) to swing too far the other way and approach all unknowns with automatic paranoia. We live in a world where we have to trust strangers every day, and if we lose that option completely then our civilization grinds to a halt.

But-- vigilance is warranted. I applaud these engineers who followed their instincts and dug into this. They all did us a huge service!

EDIT: wording, spelling

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by josephg on Friday March 29, 2024 @06:56PM (#39869918)
Yeah thanks for saying this; I agree. And as cliche as it is to look for a technical solution to a social problem, I also think better tools could help a lot here.

The current situation is ridiculous - if I pull in a compression library from npm, cargo or Python, why can that package interact with my network, make syscalls (as me) and read and write files on my computer? Leftpad shouldn’t be able to install crypto ransomware on my computer.

To solve that, package managers should include capability based security. I want to say “use this package from cargo, but refuse to compile or link into my binary any function which makes any syscall except for read and write. No open - if I want to compress or decompress a file, I’ll open the file myself and pass it in.” No messing with my filesystem. No network access. No raw asm, no trusted build scripts and no exec. What I allow is all you get.

The capability should be transitive. All dependencies of the package should be brought in under the same restriction.

In dynamic languages like (server side) JavaScript, I think this would have to be handled at runtime. We could add a capability parameter to all functions which issue syscalls (or do anything else that’s security sensitive). When the program starts, it gets an “everything” capability. That capability can be cloned and reduced to just the capabilities needed. (Think, pledge). If I want to talk to redis using a 3rd party library, I pass the redis package a capability which only allows it to open network connections. And only to this specific host on this specific port.

It wouldn’t stop all security problems. It might not even stop this one. But it would dramatically reduce the attack surface of badly behaving libraries.

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by Guvante on Friday March 29, 2024 @08:20PM (#39870521)
Doesn't this exact exploit not fixed by your capability theory?

It is hijacking a process that has network access at runtime not build time.

The build hack grabs files from the repo and inspects build parameters (in a benign way, everyone checks whether you are running on X platform etc)

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by josephg on Saturday March 30, 2024 @02:50AM (#39872439)
The problem we have right now is that any linked code can do anything, both at build time and at runtime. A good capability system should be able to stop xz from issuing network requests even if other parts of the process do interact with the network. It certainly shouldn't have permission to replace crc32_resolve() and crc64_resolve() via ifunc.

Another way of thinking about the problem is that right now every line of code within a process runs with the same permissions. If we could restrict what 3rd party libraries can do - via checks either at build time or runtime - then supply chain attacks like this would be much harder to pull off.

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by im3w1l on Saturday March 30, 2024 @04:44AM (#39872989)
I'm not convinced this is such a cure-all as any library must necessarily have the ability to "taint" its output. Like consider this library. It's a compression library. You would presumably trust it to decompress things right? Like programs? And then you run those programs with full permission? Oops..
Re:Backdoor in upstream xz/liblzma leading to SSH server com... by josephg on Saturday March 30, 2024 @05:35AM (#39873266)
It’s not a cure-all. I mean, we’re talking about infosec - so nothing is. But that said, barely any programs need the ability to execute arbitrary binaries. I can’t remember the last time I used eval() in JavaScript.

I agree that it wouldn’t stop this library from injecting backdoors into decompressed executables. But I still think it would be a big help anyway. It would stop this attack from working.

At the big picture, we need to acknowledge that we can’t implicitly trust opensource libraries on the internet. They are written by strangers, and if you wouldn’t invite them into your home you shouldn’t give them permission to execute arbitrary code with user level permissions on your computer.

I don’t think there are any one size fits all answers here. And I can’t see a way to make your “tainted output” idea work. But even so, cutting down the trusted surface area from “leftpad can cryptolocker your computer” to “Leftpad could return bad output” sounds like it would move us in the right direction.

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by Guvante on Sunday March 31, 2024 @11:12PM (#39890589)
There are attacks that embed hacks into built compilers so unless you are looking to write your software from scratch you need to trust people.

And by scratch I mean "without modern hardware" given supply chain attacks also apply to the hardware you build from.

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by josephg on Monday April 01, 2024 @10:28PM (#39901795)
Of course we need to trust people to some degree. There's an old Jewish saying - put your trust in god, but your money in the bank. I think its like that. I'm all for trusting people - but I still like how my web browser sandboxes every website I visit. That is a good idea.

We (obviously) put too much trust in little libraries like xz. I don't see a world in which people start using fewer dependencies in their projects. So given that, I think anything which makes 3rd party dependencies safer than they are now is a good thing. Hence the proposal.

The downside is it adds more complexity. Is that complexity worth it? Hard to say. Thats still worth talking about.

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by ui2RjUen875bfFA on Tuesday April 02, 2024 @04:48AM (#39903592)
i guess the big opensource community should put a little bit more trust in statistics or integrate statistic evaluation in their decission making to use specific products in their supply chains.

there are some researches on the right track already https://www.se.cs.uni-saarland.de/projects/congruence/

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by fauigerzigerk on Saturday March 30, 2024 @06:01AM (#39873386)
This approach could work for dynamic libraries, but a lot of modern ecosystems (Go, Rust, Swift) prefer to distribute packages as source code that gets compiled with the including executable or library.
Re:Backdoor in upstream xz/liblzma leading to SSH server com... by josephg on Saturday March 30, 2024 @09:34AM (#39874714)
Yes, and?

The goal is to restrict what included libraries can do. As you say, in languages like Rust, Go or Swift, the mechanism to do this would also need to work with statically linked code to work. And thats quite tricky, because there are no isolation boundaries between functions in executables.

It should still be possible to build something like this. It would just be inconvenient. In rust, swift and go you'd probably want to implement something like this at compile time.

In rust, I'd start by banning unsafe in dependencies. (Or whitelisting which projects are allowed to use unsafe code.) Then add special annotations on all the methods in the standard library which need special permissions to run. For example, File::open, fork, exec, networking, and so on. In cargo.toml, add a way to specify which permissions your child libraries get. "Import serde, but give it no OS permissions". When you compile your program, the compiler can look at the call tree of each function to see what actually gets called, and make sure the permissions match up. If you call a function in serde which in turn calls File::open (directly or indirectly), and you didn't explicitly allow that, the program should fail to compile.

It should be fine for serde to contain some utility function that calls the banned File::open, so long as the utility function isn't called.

Permissions should be in a tree. As you get further out in the dependency tree, libraries get fewer permissions. If I pass permissions {X,Y} to serde, serde can pass permission {X} to one of its dependencies in turn. But serde can't pass permission {Q} to its dependency - since it doesn't have that capability itself.

Any libraries which use unsafe are sort of trusted to do everything. You might need to insist that any package which calls unsafe code is actively whitelisted by the cargo.toml file in the project root.

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by fauigerzigerk on Sunday March 31, 2024 @06:27AM (#39882999)
>It should still be possible to build something like this. It would just be inconvenient.

Inconvenient is quite the understatement. Designing and implementing something like this for each and every language compiler/runtime requires hugely more effort than doing it on the OS level. The likelihood of mistakes is also far greater.

Perhaps it's worth exploring whether it can be done on the LLVM level so that at least some languages can share an implementation.

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by saagarjha on Saturday March 30, 2024 @09:46AM (#39874826)
Do you understand how ifuncs work? They are in the address space in the application that they run in. liblzma is resolving its own pointers!
Re:Backdoor in upstream xz/liblzma leading to SSH server com... by Bulat_Ziganshin on Saturday March 30, 2024 @12:16AM (#39871791)
if I got it right, the attack uses glibc IFUNC mechanism to patch sshd (and only sshd) to directly run some code in liblzma when sshd verifies logins.

so the problem is IFUNC mechanism, which has its valid uses but can be EASILY misused for any sort of attacks

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by josephg on Saturday March 30, 2024 @02:41AM (#39872402)
Honestly, I don't have a lot of hope that we can fix this problem for C on linux. There's just so much historical cruft in present, spread between autotools, configure, make, glibc, gcc and C itself that would need to be modified to support capabilities.

The rule we need is "If I pull in library X with some capability set, then X can't do anything not explicitly allowed by the passed set of capabilities". The problem in C is that there is currently no straightforward way to firewall off different parts of a linux process from each other. And dynamic linking on linux is done by gluing together compiled artifacts - with no way to check or understand what assembly instructions any of those parts contain.

I see two ways to solve this generally:

- Statically - ie at compile time, the compiler annotates every method with a set of permissions it (recursively) requires. The program fails to compile if a method is called which requires permissions that the caller does not pass it. In rust for example, I could imagine cargo enforcing this for rust programs. But I think it would require some changes to the C language itself if we want to add capabilities there. Maybe some compiler extensions would be enough - but probably not given a C program could obfuscate which functions call which other functions.

- Dynamically. In this case, every linux system call is replaced with a new version which takes a capability object as a parameter. When the program starts, it is given a capability by the OS and it can then use that to make child capabilities passed to different libraries. I could imagine this working in python or javascript. But for this to work in C, we need to stop libraries from just scanning the process's memory and stealing capabilities from elsewhere in the program.

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by estebarb on Saturday March 30, 2024 @10:49AM (#39875415)
Or take the Chrome / original Go approach: load that code in a different process, use some kind of RPC. With all the context switch penalty... sigh, I think it is the only way, as the MMU permissions work at a page level.
Re:Backdoor in upstream xz/liblzma leading to SSH server com... by josephg on Saturday March 30, 2024 @03:44PM (#39877951)
Firefox also has its solution of compiling dependencies to wasm, then compiling the wasm back into C code and linking that. It’s super weird, but the effect is that each dependency ends up isolated in bounds checked memory. No context switch penalty, but instead the code runs significantly slower.
Re:Backdoor in upstream xz/liblzma leading to SSH server com... by AgentME on Saturday March 30, 2024 @08:41AM (#39874306)
A process can do little to defend itself from a library it's using which has full access to its same memory. There is no security boundary there. This kind of backdoor doesn't hinge on IFUNC's existence.
Re:Backdoor in upstream xz/liblzma leading to SSH server com... by saagarjha on Saturday March 30, 2024 @09:47AM (#39874833)
The problem is that the attacker has code execution in sshd, not ifuncs
Re:Backdoor in upstream xz/liblzma leading to SSH server com... by saalweachter on Friday March 29, 2024 @07:53PM (#39870331)
Assume 3% of the population is malicious.

Enough to be cautious, enough to think about how to catch bad actors, not so much as to close yourself off and become a paranoid hermit.

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by pizzafeelsright on Friday March 29, 2024 @09:58PM (#39871140)
Huh. I never really thought of it as a percentage.

I've been evil, been wonderful, and indifferent at different stages in life.

I have known those who have done similar for money, fame, and boredom.

I think, given a backstory, incentive, opportunity, and resources it would be possible to most people to flip from wouldn't to enlisted.

Leverage has shown to be the biggest lever when it comes to compliance.

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by XorNot on Friday March 29, 2024 @10:38PM (#39871348)
It's doubtful you've been evil, or at least, you are really lacking in imagination of the true scope of what that word implies.
Re:Backdoor in upstream xz/liblzma leading to SSH server com... by mlsu on Saturday March 30, 2024 @12:32AM (#39871853)
The line between good and evil cuts through the heart of every person
Re:Backdoor in upstream xz/liblzma leading to SSH server com... by saulpw on Saturday March 30, 2024 @12:54AM (#39871935)
"Assume that 3% of the people you encounter will act maliciously."
Re:Backdoor in upstream xz/liblzma leading to SSH server com... by heresie-dabord on Saturday March 30, 2024 @07:47AM (#39873942)
We live in a time of populous, wealthy dictatorships that have computer-science expertise are openly hostile to the US and Canada.

North America is only about 5% of the world's population. [1] (We can assume that malicious actors are in North America, too, but this helps to adjust our perspective.)

The percentage of maliciousness on the Internet is much higher.

[1] _ See continental subregions. https://en.wikipedia.org/wiki/List_of_continents_and_contine...

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by tomthehero on Saturday March 30, 2024 @08:27AM (#39874191)
If you put it in other words, US and Canada are openly hostile to the rest of the world.

Bullies who would have supported Hitler had he been named Netanyahu.

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by heresie-dabord on Saturday March 30, 2024 @11:36AM (#39875845)
> If you put it in other words

You may of course choose whatever words you like. But your statement is nonsense, and the failures in the Middle East are all humanity's fault.

The kakistocracies of CN, RU, and IR are not the feeble foes of democracy and capitalism that they were in the 20th century.

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by richrichie on Saturday March 30, 2024 @11:21AM (#39875683)
Huh? The empirical evidence we have - thanks to Snowden leaks - paints a different picture. NSA is the biggest malicious actor with nearly unlimited resources at hand. They even insert hardware backdoors and intercept shipment to do that.
Re:Backdoor in upstream xz/liblzma leading to SSH server com... by heresie-dabord on Saturday March 30, 2024 @05:03PM (#39878639)
> NSA is the biggest malicious actor

I'm curious, how do you rank CN, RU, and IR?

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by richrichie on Sunday March 31, 2024 @11:16AM (#39885013)
They are all active players, but no where close to the top dog.
Re:Backdoor in upstream xz/liblzma leading to SSH server com... by rieter on Monday April 01, 2024 @10:01PM (#39901607)
That doesn't mean that the entire population of those countries is actively hostile to you.
Re:Backdoor in upstream xz/liblzma leading to SSH server com... by account42 on Wednesday April 03, 2024 @08:19AM (#39916488)
The US government itself is openly hostile to the US (as well as to the rest of the world).

> The percentage of maliciousness on the Internet is much higher.

A baseless assumption.

Re:Backdoor in upstream xz/liblzma leading to SSH server com... by heresie-dabord on Friday March 29, 2024 @07:57PM (#39870363)
Threat actors create personas. We will need strong social trust to protect our important projects and dependencies.
Re:Backdoor in upstream xz/liblzma leading to SSH server com... by account42 on Wednesday April 03, 2024 @08:01AM (#39916321)
> We, particularly those us who work on critical infrastructure or software

We should also be asking ourselves if we are working on critical infrastructure. Lasse Collin probably did not consider liblzma being loaded by sshd when vetting the new maintainer. Did the xz project ever agree to this responsibility?

We should also be asking ourselfs if each dependency of critical infrastructure is worth the risk. sshd linking libsystemd just to write a few bytes into an open fd is absurd. libsystemd pulling in liblzma because hey it also does compressed logging is absurd. Yet this kind of absurd dependency bloat is everywhere.