That's a strong indication it's targeting sshd specifically.

"There's lots of stuff I have not analyzed and most of what I observed is purely from observation rather than exhaustively analyzing the backdoor code."

"There are other checks I have not fully traced."

I did a quick diff of the source (.orig file from packages.ubuntu.com) and the content mostly matched the 5.4.5 github tag except for Changelog and some translation files. It does match the tarball content, though.

So for 5.4.5 the tagged release and download on github differ.

It does change format strings, e.g.

   +#: src/xz/args.c:735
   +#, fuzzy
   +#| msgid "%s: With --format=raw, --suffix=.SUF is required unless writing to stdout"
   +msgid "With --format=raw, --suffix=.SUF is required unless writing to stdout"
   +msgstr "%s: amb --format=raw, --suffix=.SUF és necessari si no s'escriu a la sortida estàndard"

[Edit] formatting

Anyway, so... the xz project has been compromised for a long time, at least since 5.4.5. I see that this JiaT75 guy has been the primary guy in charge of at least the GitHub releases for years. Should we view all releases after he got involved as probably compromised?

My TLDR is that I would regard all commits by JiaT75 as potentially compromised.

Given the ability to manipulate gitnhistory I am not sure if a simple time based revert is enough.

It would be great to compare old copies of the repo with the current state. There is no guarantee that the history wasn't tampered with.

Overall the only safe action would IMHO to establish a new upstream from an assumed good state, then fully audit it. At that point we should probably just abandon it and use zstd instead.

Xz is an implant of 7zip's LZMA(2) compression into traditional Unix archiver skeleton. It trades long compression times and giant dictionaries (that need lots of memory) for better (“much-better-than-deflate”) compression ratios. Therefore, zstd, no matter how fashionable that name might be in some circles, is not a replacement for xz.

It should also be noted that those LZMA-based archive formats might not be considered state-of-the-art today. If you worry about data density, there are options for both faster compression at the same size, and better compression in the same amount of time (provided that data is generally compressible). 7zip and xz are widespread and well tested, though, and allow decompression to be fast, which might be important in some cases. Alternatives often decompress much slowly. This is also a trade-off between total time spent on X nodes compressing data, and Y nodes decompressing data. When X is 1, and Y is in the millions (say, software distribution), you can spend A LOT of time compressing even for relatively minuscule gains without affecting the scales.

It should also be noted that many (or most) decoders of top compressing archivers are implemented as virtual machines executing chains of transform and unpack operations defined in archive file over pieces of data also saved there. Or, looking from a different angle, complex state machines initializing their state using complex data in the archive. Compressor tries to find most suitable combination of basic steps based on input data, and stores the result in the archive. (This is logically completed in neural network compression tools which learn what to do with data from data itself.) As some people may know, implementing all that byte juggling safely and effectively is a herculean task, and compression tools had exploits in the past because of that. Switching to a better solution might introduce a lot more potentially exploited bugs.

You should use ultra settings and >=19 as the compression level. E.g. arch used 20 and higher compression levels do exist, but they were already at a <1% increase.

It does beat xz for these tasks. It's just not the default settings as those are indeed optimized for the lzo to gzip/bzip2 range.

Zstd does reach LZMA compression ratios on high levels, but compression times also drop to LZMA level. Which, obviously, was clearly planned in advance to cover both high speed online applications and slow offline compression (unlike, say, brotli). Official limit on levels can also be explained by absence of gains on most inputs in development tests.

Distribution packages contain binary and mixed data, which might be less compressible. For text and mostly text, I suppose that some old style LZ-based tools can still produce an archive roughly 5% percent smaller (and still unpack fast); other compression algorithms can certainly squeeze it much better, but have symmetric time requirements. I was worried about the latter kind being introduced as a replacement solution.

bzip2 is a pig that has no place being in the same sentence as lzo and gzip. It's nieche was maximum compression no matter the speed but it hasn't been relevant even there for a long time.

Yet tools still need to support bzip2 because bzip2 archives are still out there and are still being produced. So we can't get rid of libbz2 anytime soon - same for liblzma.

found it mentioned in https://github.com/facebook/proxygen/blob/main/build/fbcode_..., looks like it's going to be cousin of zstd, but maybe for the stronger compression use cases

Rewritten history is not a real concern because it would have been immediately noticed by anyone updating an existing checkout.

> Overall the only safe action would IMHO to establish a new upstream from an assumed good state, then fully audit it. At that point we should probably just abandon it and use zstd instead.

This is absurd and also impossible without breaking backwards compatibility all over the place.

So it's just odd that the tags and release tarballs diverge.

I'm surprised .deb doesn't have a better approach. RPM has epoch for this purpose http://novosial.org/rpm/epoch/index.html

Two reasons:

1. Once you bump the epoch, you have to use it forever. 2. The deb filename often doesn't contain the epoch (we use a colon which isn't valid on many filesystems), so an epoch-revert will give the same file name as pre-epoch, which breaks your repository.

So, the current best practice is the +really+ thing.

And I fail to see why bumping the epoch would ever be a problem. Using the epoch not a reason why its bad.

So now I actually bothered to look it up, and it turns out the actual reason is that the epoch changes what version is considered "greater", but it's not part of the .deb filename, so you still can't reuse version numbers used in the past. If you release 5.0, then 5.1, then you want to rollback and release 1:5.0, it's going to break things in the Debian archives. https://www.debian.org/doc/debian-policy/ch-binary.html#uniq...

Additionally, once you add an epoch you're stuck with it forever, while if you use 5.1+really5.0, you can get rid of the kludge when 5.2 is out. https://www.debian.org/doc/debian-policy/ch-controlfields.ht...

Maybe they’re expecting a 5.6.x release shortly that fixes all these issues & don’t want to add an epoch for a very short term packaging issue?

I read through the report, but what wasn't directly clear to me was: what does the exploit actually do?

My normal internet connection has such an appalling upload that I don't think anything relevant could be uploaded. But I will change my ssh keys asap.

Possible but unlikely.

> I read through the report, but what wasn't directly clear to me was: what does the exploit actually do?

It injects code that runs early during sshd connection establishment. Likely allowing remote code execution if you know the right magic to send to the server.

With our current knowledge, stable shouldn’t be affected by this.

> Running OpenSSH sshd from systemd

I think this is irrelevant.

From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.

Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.

It looks like the backdoor "deactivates" itself when it detects being started interactively, as a security researcher might. I was eventually able to circumvent that, but unless you do so, it'll not be active when started interactively.

However, the backdoor would also be active if you started it with an shell script (as the traditional sys-v rc scripts did) outside the context of an interactive shell, as TERM wouldn't be set either in that context.

> Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.

There's no xz code in openssh.

OpenSSH is developed by the OpenBSD project, and systemd is not compatible with OpenBSD. The upstream project has no systemd or liblzma code to strip. If your sshd binary links to liblzma, it's because the package maintainers for your distro have gone out of their way to add systemd's patch to your sshd binary.

> From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.

From what I understand, the backdoor detects if it's in any of a handful of different debug environments. If it's in a debug environment or not launched by systemd, it won't hook itself up. ("nothing to see here folks...") But if sshd isn't linked to liblzma to begin with, none of the backdoor's code even exists in the processes' page maps.

I'm still downgrading to an unaffected version, of course, but it's nice to know I was never vulnerable just by typing 'ldd `which sshd`' and not seeing liblzma.so.

liblzma5:amd64 5.4.1-0.2

Ironic considering security is often advertised as a feature of rolling release distros. I suppose in most instances it does provide better security, but there are some advantages to Debian's approach (stable Debian, that is).

Security is a feature of rolling release. But supply-chain attacks like this are the exception to the rule.

This particular backdoor is not shipped inside of a security patch, right?