This comes up a lot, but how often do you end up in a scenario where there's a critical security hole and you _can't_ patch it because one program somewhere is incompatible with the new version? Maybe even a program that isn't security critical.
Plus what you mentioned about testing. If you update each application individually, you can do it piecemeal. You can test your critical security software and roll out a new version immediately, and then you can test the less critical software second. In some systems you can also do partial testing, and then do full testing later because you're confident that you have the ability to roll back just one or two pieces of software if they break.
It's the same amount of work, but you don't have to wait for literally the entire system to be stable before you start rolling out patches.
I don't think it's 100% a bad idea to use shared libraries, I do think there are some instances where it does make sense, and centralized interfaces/dependencies have some advantages, particularly for very core, obviously shared, extremely stable interfaces like Linus is talking about here. But it's not like these are strict advantages and in many instances I suspect that central fixes can actually be downsides. You don't want to be waiting on your entire system to get tested before you roll out a security fix for a dependency in your web server.
Talking about user usecases, every time I play a game on Steam. At the very least there are GNU TLS versioning problems. That's why steam packages it's own library system containing multiple versions of the same library -- thus rendering all of the benefits of shared libraries completely null.
One day, game developers will package static binaries, and the compiler will be able to rip out all of the functions that they don't use, and I won't have to have 5 - 20 copies of the same library on my system just sitting around -- worse if you have multiple steam installs in the same /home partition because you're a distro hopper.
One day...
These are such different use cases that I think completely different standards and processes as well as build systems are going to become the norm for big critical infrastructure versus what is running on your favorite laptop.
So it's actually less wasteful.
Link-time optimization is already a thing. It's sad it doesn't seem to be used more often.
On the other hand, if I am static linking, then there are no other programs that could use the static library. (Or, rather, if they do, they have their own embedded copy.) The LTO is free to remove any functions that I don't need, reducing the total amount that I need to ship.
LTO is really a much bigger deal, the main reason to use it is not throwing unused stuff away, but inlining across all the things.
Wouldn't you still need at least runtime linking (dlopen) to link to OpenGL/Vulkan/etc.?
Some libraries (OpenGL, Vulkan, ALSA, ..) the shared library provides the lowest stable cross-hardware interface there is so linking the library makes no sense.
That’s not the point. The point is having to find and patch multiple copies of a library in case of vulnerability instead of just one.
Giving up the policy to enforce shared libraries would just make the work of security teams much harder.
But it doesn't really matter. What matters is that whatever system is in use needs to have a control system that can quickly and reliably tell you everything that uses a vulnerable library version, and can then apply the fixes where available and remind you of the deficiencies.
That metadata and dependency checking can be handled in any number of ways, but if it's not done, you are guaranteed not to know what's going on.
If a library is used inside a unikernel, inside a container, inside a virtual machine, inside venvs or stows or pex's or bundles, the person responsible for runtime operations needs to be able to ask what is using this library, and what version, and how can it be patched. Getting an incomplete answer is bad.
But there's one other advantage of the shared library thing, which is that when you need to react fast (to a critical security vulnerability, for example), it is possible to do it without coordinating with N number of project/package maintainers and getting them all to rebuild.
You still do want to coordinate (at least for testing purposes), but maybe in emergencies it's more important to get a fix in place ASAP.
...and then you deploy and discover that somebody was depending on that "unwanted" side effect.
I fully agree with that, didn't understand the rest.
Now it's a month later. What do you need in ongoing operations, assuming you want to keep providing reasonable security?
You need to know when any of your dependencies makes a security-related change, and then you need to evaluate whether it affects you.
You need to know which systems in your service are running which versions of that dependency.
You need to be able to test the new version.
You need to be able to deploy the new version.
It doesn't matter what your underlying paradigm is. Microservices, unikernels, "serverless", monoliths, packages, virtual machines, containers, Kubernetes, OpenStack, blah blah blah. Whatever you've got, it needs to fulfill those functions in a way which is effective and efficient.
The problem is that relatively few such systems do, and of those that do, some of them don't cooperate well with each other.
It's plausible that you have operating system packages with a great upstream security team, so you get new releases promptly... and at the same time, you use a bunch of Python Packages that are not packaged by your OS, so you need to subscribe individually to each of their changefeeds and pay attention.
Does that help?
The whole "shared libraries are better for security" idea is basically stuck in a world where we don't really have proper dependency management and everyone just depends on "version whatever".
This is interestingly also holding back a lot of ABI breaking changes in C++ because people are afraid it'll break the world... which, tbf, it will in a shared library world. If dependencies are managed properly... not so much.
I wish distros could just go back to managing applications rather than worrying so much about libraries.
EDIT: There are advantages with deduplicating the memory for identical libraries, etc., but I don't see that as a major concern in today's world unless you're working with seriously memory-constrained systems. Even just checksumming memory pages and deduplicating that way might be good enough.
And in case your definition of proper dependency management is 'stricter', then you simply state that you depend on a vulnerable version, and fixing the issue will be far more cumbersome as it requires manual intervention as well, instead of an automated update and rebuild.
If it is looser, then it will also be far more cumbersome, as you have to watch out for breakage when trying to rebuild, and you need to update your program for the new API of the library before you can even fix the issue at all.
In practice it works: see autopkgtest and https://ci.debian.net, which reruns the tests for each reverse dependency of a package, every time a library gets updated.
I know for a fact that other commercial, corporate-backed distributions are far away from that sophistication, though.
No, they're not. Both Fedora/RHEL and openSUSE/SLE do similar checks. Every update submission goes through these kinds of checks.
Heck, the guy who created autopkgtest works at Red Hat and helped design the testing system used in Fedora and RHEL now. openSUSE has been doing dependency checks and tests for almost a decade now, with the combination of the openSUSE Build Service and OpenQA.
No, rebuilding is way heavier.
Even worse, some languages insist on building against fixed version of their dependencies.
It forces distributions to keep multiple version of the same library and this leads to a combinatorics explosion of packages to fix, backport, compile and test.
It's simply unsustainable and it's hurting distributions already.
- statically link a library
- have all binaries in the system use the same version of the library, and updated when the library needs to be updated.
It's worse if the program that needs the update also is broken by the update, of course.
Now it's your choice... you either lose B but protect the rest of the infrastructure from hackers... or you think the CVE doesn't apply to your usecase (internal thing on a testing server), and don't upgrade L to keep B working.
You can also install both versions of L. You can also patch the broken part of L out at the old version, if it's not mission critical. There's a lot of things you can do.
Having one giant binary file with everything statically compiled in is worse in every way, except for distribution-as-a-single-file (but you can already do this now, by putting the binary and the libraries in a single zip, dump everytinh in /opt/foo, and let user find the vulnerable library manually... which again, sucks.
Installing both versions of L is usually hard. It's one thing if it's OpenSSL[1] 1.1 vs 1.0, but if 1.0.0e is needed for security and 1.0.0d is needed for other applications, how do you make that work in Debian (or any other system that's at least somewhat mainstream)?
[1] Not to pick on OpenSSL, but it's kind of the poster child for important to pick up updates that also break things; but at least they provide security updates across branches that are trying to stay compatible.
Now with Docker, it becomes, find all the Docker containers that have that shared library, which may even be a bigger pain.
Not all uses, but it seems as if a significant use of Docker is just a a very complicated replacement for a single statically linked binary.
Imagine statically linking openssl having to rebuild tenths of thousands of packages every time there's an update!
I worked these systems in various large organization and distros. We did the math.
On top of that, delivering frequent very large updates to deployed systems is very difficult in a lot of environments.
Debian security issues thousands patched shared libraries every year without testing them against every program, and without causing failures. They do that by back porting the security fix to the version of the library Debian uses.
I gather you are a developer (as am I), and I'm guessing that scenario didn't occur to you as no developer would do it. But without it Debian possibly wouldn't be sustainable. There are 21,000 packages linked against libc in Debian. Recompiling all of them when a security problem happens in libc may well be beyond the resources Debian has available.
In fact while it's true backward compatibility can't be trusted for some libraries, it can for many. That's easily verified - Debian ships about 70,000 packages, yet typically Debian ships just one version of each library. Again the stand out example is libc - which is why Debian can fearlessly link 21,000 packages again the same version.
I'm guess most of the people here criticising shared libraries are developers, and it's true shared libraries don't offer application developers much. But they weren't created by application developers or for application developers. They were created by the likes of SUN and Microsoft, would wanted to skip one WIN32.DLL so they could update just that when they found a defect in it. In Microsoft's case recompiling every program that depended on it was literally impossible.
On Linux it the only thing it should change is that if a vulnerability is discovered let's say in OpenSSL all the software that depends on OpenSSL must be updated, and that could be potentially half the binaries of your system. It's a matter of download size, that in reality doesn't matter that much (and in theory can be optimized with package managers that applies binary patches to packages).
It's the maintainers of the distribution that notes the vulnerability in OpenSSL and decides to rebuild all packages that are statically linked to the vulnerable library. But for the final user the only thing he has to do is still an `apt upgrade` or `pacman -Syu` or whatever, and he would still get all the software patched.
Actually now that I think about it, building by yourself might put you at a disadvantage here, as you'd have to remember to rebuild everything. I'm kinda lazy when it comes to updates so not sure if I like the idea anymore with having to rebuild all the software I built myself lol, but it probably could be solved by also shipping .so versions of libraries.
I don't have the best solution for it, but the current trend I see on Hacker News of supporting automatic updates everywhere, sometimes without even giving users an opt-out let alone an opt-in, is rather alarming.
I'm just throwing ideas around so you should definitely take what I'm saying with a grain of salt. It just would be interesting to see a distro like that and see what the downsides of this solution are. Chances are that there probably already is something like this and I'm just not aware of it and I'm reinventing the wheel.
I'm in the same boat, as somebody who isn't in the security field. I try to keep up with it, but will occasionally comment on things that I don't understand.
On stable production systems? Never.
The CIP project https://www.cip-project.org/ aims to backport security fixes to Linux for *25* years after each CIP release.
No sensible organization runs bleeding edge OSes on their airplanes, power plants, payment processors, industrial plants...
Most distros dislike doing so, and scale it back as soon as it becomes technically possible.
Or is that step just not happening?
I just feel like, one of the big points of a package manager is that I can look up "what is program X's dependencies, and what packages rely on it?"
For overlapping memory, use memmove()
Any code that invokes UB is not a valid C program, regardless of implemention.
More practically, ignoring the bug that did happen, libc also has multiple implementations of these functions and picks one based on HW it is running on. So even a statically linked glibc could behave differently on different HW. Always read the docs, this is well defined in the standard.
> Any code that invokes UB is not a valid C program, regardless of implemention.
I disagree. UB is defined (3.4.3) merely as behavior the standard imposes no requirements upon. The definition does not preclude an implementation from having reasonable behavior for situations the standard considers undefined.
This nuance is very important for the topic at had because many programs are written for specific implementations, not specs, and doing this is completely reasonable.
Did glibc change the behavior of memcpy (the ABI symbol) or memcpy (the C API symbol) which currently maps to the memcpy@GLIBC_2.14 ABI symbol?
- shared libs
- name indirection (dns domains, css classes)
The theory is as follows: If a security flaw (or other serious bug) is found, then you can fix it centrally if you use shared libraries, as opposed to finding every application that uses the library and updating each separately.
In practice this doesn't work because each application has to be tested against the new version of the library.