> I know for a fact that other commercial, corporate-backed distributions are far away from that sophistication, though.

No, they're not. Both Fedora/RHEL and openSUSE/SLE do similar checks. Every update submission goes through these kinds of checks.

Heck, the guy who created autopkgtest works at Red Hat and helped design the testing system used in Fedora and RHEL now. openSUSE has been doing dependency checks and tests for almost a decade now, with the combination of the openSUSE Build Service and OpenQA.

Ex SUSE employee here. OBS and OpenQA are nowhere close to Debian CI.

Having worked with both systems, Debian CI does not focus on system integration. Fedora CI and Debian CI are more similar than different, but Fedora also has an OpenQA instance for doing the system integration testing as openSUSE does. openSUSE's main weakness is that they don't do deeper inspections of RPM artifacts, the dependency graph, etc. They don't feel they need it because OBS auto-rebuilds everything on every submission anyway. The Fedora CI tooling absolutely does this since auto-rebuilds on build aren't a thing there, and it's done on PRs as well as update submissions.

If you can retest everything, you can rebuild everything. But that doesn't help the network bandwidth issue. Debian doesn't have incremental binary diff packages anyway (delta debs) in the default repos anyway, so there's room for improvement there.

> If you can retest everything, you can rebuild everything

No, rebuilding is way heavier.

Even worse, some languages insist on building against fixed version of their dependencies.

It forces distributions to keep multiple version of the same library and this leads to a combinatorics explosion of packages to fix, backport, compile and test.

It's simply unsustainable and it's hurting distributions already.

Distros have been known to patch packages that wanted a fixed version of a dependency. In fact, most have it has their default policy.

Yes, that's my point.

To rephrase what I meant; those two aren't exclusive:

- statically link a library

- have all binaries in the system use the same version of the library, and updated when the library needs to be updated.

So if the CVE says that you need to update library L, and program A uses the thing that's broken and it's ok to update, but program B doesn't use the broken thing, but the L upgrade breaks it, CI would let you know that you're screwed before you upgrade... but you're still screwed.

It's worse if the program that needs the update also is broken by the update, of course.

So, what's the alternative?

Now it's your choice... you either lose B but protect the rest of the infrastructure from hackers... or you think the CVE doesn't apply to your usecase (internal thing on a testing server), and don't upgrade L to keep B working.

You can also install both versions of L. You can also patch the broken part of L out at the old version, if it's not mission critical. There's a lot of things you can do.

Having one giant binary file with everything statically compiled in is worse in every way, except for distribution-as-a-single-file (but you can already do this now, by putting the binary and the libraries in a single zip, dump everytinh in /opt/foo, and let user find the vulnerable library manually... which again, sucks.

If it were static libraries, you'd upgrade the package for A (which would need to be recompiled with updated L) and leave B alone. As a low priority followup, fix either B or L so they can work together again (or wait for someone else to fix and release).

Installing both versions of L is usually hard. It's one thing if it's OpenSSL[1] 1.1 vs 1.0, but if 1.0.0e is needed for security and 1.0.0d is needed for other applications, how do you make that work in Debian (or any other system that's at least somewhat mainstream)?

[1] Not to pick on OpenSSL, but it's kind of the poster child for important to pick up updates that also break things; but at least they provide security updates across branches that are trying to stay compatible.