I like the theory that actually, it wasn’t luck but was picked up on by detection tools of a large entity (Google / Microsoft / NSA / whatever), and they’re just presenting the story like this to keep their detection methods a secret. It’s what I would do.

I doubt that if Google detected it with some internal tool, they'd reach out to Microsoft to hide their contribution.

It was reported by an MS engineer who happens to be involved in another OSS project. MS is doing business with the US intelligence community, for example there is the Skype story: First, rumors that NSA offers a lot of money for people who can break Skype's E2E encryption, then MS buys Skype, then MS changes Skype's client to not be E2E encrypted any more and to use MS servers instead of peer to peer, allowing undetectable wiretapping of arbitrary connections.

But it's a quite credible story too that it was just a random discovery. Even if it was the NSA, why would they hide that capability. It doesn't take much to run a script to compare git state with uploaded source tarballs in distros like Debian (Debian has separate tarballs for the source and the source with Debian patches applied).

> It was reported by an MS engineer who happens to be involved in another OSS project.

I view it as being OSS or postgresql dev that happens to work at microsoft. I've been doing the former for much longer (starting somewhere between 2005 and 2008, depending on how you count) than the latter (2019-12).

Thanks for the explanation. Also thanks for catching this and protecting us all; I think in the end it's way more believable that you indeed found it on your own, above was just brainless blathering into the ether =). Lastly, thanks for your Postgres contributions.

Just wanted to say thank you for your work and attention to detail, it's immensely valuable and we're all very grateful for it.

“Even if it was the NSA, why would they hide that capability”

Perhaps you’re not familiar with what NSA historically stood for: Never Say Anything.

Googling for "site:nsa.gov filetype:pdf" gives tens of thousands of results with documents produced by the NSA about various things (the google counter is known to lie but that's not my point). They do publish things.

Intelligence agencies are very careful about sharing their findings, even with "friends", because the findings will disclose some information about their capabilities and possibly methods.

Let's say agency A has some scanning capability on open source software that detected this backdoor attempt by agency B. If they had gone public, agency B now knows they have this ability. So agency B will adjust their ways the next time and the scanning capability becomes less useful. While if agency A had told Microsoft to "find" this by accident, nobody would know about their scanning capability. And the next attempt by agency B would only try to avoid having the performance impact this first attempt had, probably leaving it visible to agency A.

> an MS engineer

no this engineer is world-known for being core PostgreSQL, a team with high standards.. unlike that company you mention

Yeah definitely, good point. I got that wrong above: He is in fact a postgres contributor who happens to be MS employee and not an MS employee who happens to be a postgres contributor.

He is also one of the sharpest developers I have had the fortune to speak to (and do some minor work with on the mailing list) and he knows a lot about performance and profiling. I 100% think that he found it on his own. He would also be an odd choice for Microsoft to pick since I doubt he works anywhere close to any of their security teams (unless they went through some serious lengths to find just the guy where it would be 100% believable that he just stumbled on it).

Sorry for that ugly comparison, but that explanation reminds me of the theories when covid started, that it was created by secret organization that is actually ruling the world.

People love when there's some explanation that doesn't involve randomness, because with randomness looks like we don't have grasp on things.

Google actually had tooling that was detecting it, but he disabled check that would show it.

Google/Microsoft/NSA could just say they detected it with internal tooling and not disclose how exactly. Google and Microsoft would love to have credit.

They could announce it without revealing their detection method. I don't see what the extra indirection buys them.

Shutting down 99.9% of speculation on how the vulnerability was found in the first place.

(1) obviously not

(2) they could just say they found it during some routine dependency review or whatever

I mean, yes. I’ve read all the commentary I can get my hands on about this incident because it is fascinating and this is the first instance of some parallel construction theory of finding it I’ve seen.

Second, maybe a routine dependency review is how they _actually_ found it but they don’t want future people like this focusing too much on that otherwise they may try to mitigate, whereas now they may focus on something inane like a 0.5 second increase in sshd’s load time or whatever.

The attacker changed the projects contact details at oss fuzz (an automated detection tool). There’s an interesting discussion as to whether that would have picked up the vulnerability https://github.com/google/oss-fuzz/issues/11760

I work on oss-fuzz.

I don't think it's plausible OSS-Fuzz could have found this. The backdoor required a build configuration that was not used in OSS-Fuzz.

I'm guessing "Jia Tan" knew this and made changes to XZ's use of OSS-Fuzz for the purposes of cementing their position as the new maintainer of XZ, rather than out of worry OSS-Fuzz would find the backdoor as people have speculated.

How many oss-fuzz packages have a Dockerfile that runs apt-get install liblzma-dev first?

Had this not been discovered, the backdoored version of xz could have eventually ended up in the ubuntu version oss-fuzz uses for its docker image - and linked into all those packages being tested as well.

Except now there's an explanation if fuzzing starts to fail - honggfuzz uses -fsanitize which is incompatible with xz's use of ifunc, so any package that depends on it should rebuild xz from source with --disable-ifunc instead of using the binary package.

This is interesting, but do you think this would have aroused enough suspicion to find the backdoor (after every Ubuntu user was owned by it)? I don't see why this is the case. It wasn't a secret that ifuncs were being used in XZ.

And if that's the case, it was sloppy of "Jia" to disable it in OSS-Fuzz and not do this:

``` __attribute__((__used__,__no_sanitize_address__)) ```

to the XZ source code to fix the false positive and turn off the compilation warning, no attention would have been drawn to this at all since no one would have to change their build script.

With or without this PR, it's very unlikely OSS-Fuzz would have found the bug. OSS-Fuzz also happens to be on Ubuntu 20. I'm not very familiar with Ubuntu release cycles, but I think it would have been a very long time before backdoored packages made their way into Ubuntu 20.

That's a fascinating extra detail. They really tried to cover all their bases.

There's some plausible evidence here that they may've tried to use alter egos to encourage Debian to update the package: https://twitter.com/f0wlsec/status/1773824841331740708

> ... nd they’re just presenting the story like this to keep their detection methods a secret. It’s what I would do.

Basically "parallel construction". It's very possible it's what happened.

https://en.wikipedia.org/wiki/Parallel_construction

It's really interesting to think what might've happened if they could've implemented this with much less performance overhead. How long might it have lasted for? Years?

You're describing parallel construction and it is something that happens all of the time.

While we are speculating, there was this case of university students trying to introduce malicious commits in the kernel in order to test open source to such attack vectors. Perhaps this was similar "research" by some students.

https://www.cnx-software.com/2021/04/22/phd-students-willful...

It'd make total sense if it was. This way you get to have the backdoor without your enemies being able to use it against your own companies.

I'm not sure why everyone is 100% sure this was a state-sponsored security breach. I agree that it's more likely than not state-sponsored, but I can imagine all sorts of other groups who would have an interest in something like this, organized crime in particular. Imagine how many banks or crypto wallets they could break into with a RCE this pervasive.

Especially considering this introduced a 500ms waiting time. But surely this was quite a risky time investment, 2 years. How likely is it that this was the only attempt if this was done by a group? (And maybe there were failed attempts after trying to take over maintenance of other packages?) Maybe really a very well-funded cybercrime group that can afford such moonshot endeavours or a state group that doesn't completely know yet what it's doing or isn't that well equipped (anymore?). I'm definitely curious about analysis of attribution

Motive and patience. Motive as you point out is shared by many parties.

Typically its only state agencies that will fund an operation with uncertain pay off over long periods of time. That type of patience is expensive.

Online criminals are beholden to changing market pressures and short term investment pressures like any other start up.

I'm not 100% sure - it could have been a criminal(s) or even a single motivated actor.

That said it's a lot of work - 2 years at least. It's an exploit that's so good that you'd have to use incredibly carefully - also because if/when it's discovered it's going to break everywhere.

I've read descriptions about how the NSA (and presumably other such agencies) and they're really careful. The first job is to make sure the target doesn't get confirmation that they are in fact a target. The second is that they always cover their tracks so the target doesn't know they were a target.

Criminals tend to do the first but almost never the second so a tool like this - while I'm sure they would love - isn't worth the amount of work it would take to develop.

Again - I'm not 100% on this but ... 40% ? say 20% criminals, 10% lone wolf?

I read someome speculating that the performance issue was intentional, so infected machines could be easily identified by an internet wide scan without arousing further suspicicion.

If this is or becomes a widespread method, then anti-malware groups should perhaps conduct these scans themselves.

Very small differences in performance can be detected over the network as long as you have enough samples. Given that every port 22 is being hit by a gazillion attempts per day already, sample count shouldn’t be an issue.

So if distinguishing infected machines was their intention they definitely over-egged it.

Do we have a detailed technical analysis of the code? I read a few analysis but they all seem preliminary. It is very useful to learn from the code.

There's a few links down at the bottom of the OP to quite detailed analysis. From there you could join a Discord where discussion is ongoing.

Thanks coldpie.

I don't think it was executed incredibly well. There were definitely very clever aspects but they made multiple mistakes - triggering Valgrind, the performance issue, using a `.` to break the Landlock test, not giving the author a proper background identity.

I guess you could also include the fact that they made it a very obvious back door rather than an exploitable bug, but that has the advantage of only letting you exploit it so it was probably an intentional trade-off.

Just think how many back doors / intentional bugs there are that we don't know about because they didn't make any of these mistakes.

Maybe it's the first successful attempt of a state which nobody would right now suspect as capable of carrying this out. Everyone is looking at the big guys but a new player has entered the game.

Yeah, we should probably expect that there are roughly 1/p(found) more of these lurking out there. Not a pleasant thought.

one question I still have is what exactly the performance issue was? I heard it might be related to enumeration of shared libraries, decoding of the scrambled strings[1], etc. anyone know for sure yet?

one other point for investigation is if the code is similar to any other known implants? like the way it obfuscates strings, the way it detects debuggers, the way its setting up a vtable, there might be code fragments shared across projects. Which might give clues about its origin.

[1] https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01

It's the symbol lookup that's slow.

why does the backdoor rely on symbol lookup? is it because of the use of IFUNC ?

I believe it is looking for functions in OpenSSH that do key validation etc

Was the performance issue pure luck? Or was it a subtle bit of sabotage by someone inside the attacking group worried about the implications of the capability?

If it had been successfully and secretly deployed, this is the sort of thing that could make your leaders much more comfortable with starting a "limited war".

There are shades of "Setec Astronomy" here.

Plot twist:

It was a psyop to increase the scrutiny around OSS components.

Kidding. Mostly...

But given the amount of scrutiny folks are going to start putting into some supply chains... Probably cheaper to execute than most company's annual security awareness budgets cost.

Considering how difficult it might be (and identifiable) to attempt direct exploitation of this without being sure your target is vulnerable, it’s plausible the performance issue allowed for an identifiable delay in attempts. This might be useful in determining whether to attempt the exploit, with an auto-skip if it received a response in less than N milliseconds.