Oh, yes, I certainly would. Shared libraries are an expensive, trouble-prone solution for that relatively uncommon problem.

Anyway that's not why we have shared libraries. They were invented so that memory managers could load the same library once, then map it into the address space for every process which used it. This kind of savings mattered back when RAM was expensive and executable code represented a meaningful share of the total memory a program might use. Of course that was a very long time ago now.

> Oh, yes, I certainly would. Shared libraries are an expensive, trouble-prone solution for that relatively uncommon problem.

This personal assertion has no bearing in the real world. Please do shine some light on what exactly do you interpret as "expensive", and in the process explain why do you feel that basic software design principles like modularity and the ability to fix security vulnerabilities in the whole system by updating a single shared library is "trouble-prone".

I mean, the whole world never saw a real world problem, not the UNIX comunity nor Microsoft and it's Windows design nor Apple and it's macOS design not anyone ever, but somehow here you are claiming the opposite.

> the whole world never saw a real world problem, not the UNIX comunity nor Microsoft and it's Windows design

"DLL hell" [1] was absolutely a thing. It was solved only by everyone shipping the world in their application directories.

[1]: https://en.wikipedia.org/wiki/DLL_Hell

Not to mention dynamic linking is brutally slow. I don't think people even realize how many cycles are wasted. The notion that dynamic linking never had any real problems and still doesn't is totally false.

> basic software design principles like modularity

A single product can have a modular design, but modularity across unrelated products designed by different designers is a different concept that should have a different name.

In my opinion it’s just not something that really works. This is similar to the “ability” of some software to have multiple pluggable database engines. This always yields a product that can use only the lowest common denominator… badly.

But now imagine having to coordinate the sharing of database engines across multiple relying products! So upgrading Oracle would break A, B and fix C, but switching A and B to MySQL would require and update to MySQL that would break D, E and F.

That kind of thing happens all the time and people run away screaming rather than deal with it.

Shared libraries are the same problem.

You aren't updating it for the whole system if some of the binaries are statically linked or have their own implementations. The point is that no matter what, you still have to audit all binaries.

Also: text pages are shared in the typical case. It was largely a spurious argument to begin with. If it didn't fuck over `musl` et. al. and `libstdc++` it would have been relegated to corner cases years ago.

Is your first sentence missing a negation, or can you elaborate?

I should have been more precise, but that can become a treatise by HN comment standards easily so consider there to be an asterisk next to all of this for myriad platform/toolchain corner cases.

To the extent that shared libraries economize on memory, or are claimed to, it's when a lot of different processes link the same version of the same library, e.g. your `libc` (or the original motivating example of X). The tradeoff here is that the kernel has an atomic unit of bringing data into RAM (AKA "core" for the old-timers), which is usually a 4kb page, and shared libraries are indivisible: you use `memcpy` and you get it and whatever else is in the 4kb chunk somebody else's linker put it in. Not a bad deal for `memcpy` because it's in everything, but a very bad deal for most of the symbols in e.g. `glibc`.

The counter-point is that with static linking, the linker brings in only the symbols you use. You do repeat those symbols for each `ELF`, but not for each process. This is often (usually?) a better deal.

A little poking around on my system to kind of illustrate this a bit more viscerally: https://pastebin.com/0zpVqA0r (apologies for the clumsy editing, it would be too long if I left in every symbol from e.g. `glibc`).

Admittedly, that's a headless machine, and some big clunky Gnome/Xorg thing might tell a different story, but it would have to be 100-1000x worse to be relevant in 2022, and almost certainly wouldn't offset the (heinous) costs. And while this is speculation, I think it would be negative worse, because there just aren't that many distinct `ELF`'s loaded on any typical system even with a desktop environment. Chromium or whatever is going to be like, all your RAM even on KDE.

There are couple of great CppCon talks that really go deep here (including one by Matt Godbolt!): https://www.youtube.com/watch?v=dOfucXtyEsU, https://www.youtube.com/watch?v=xVT1y0xWgww.

Don't be put off by the fact it's CppCon, all your GNU stuff is the same story.

Uncommon by all uses of shared libraries maybe, but certainly not uncommon for security-oriented libraries. A couple times per year is WAY more than often enough to be worth shared libraries, as otherwise you're talking about rebuilding much of the world multiple times per year per security-oriented library.

And nearly all libraries eventually become "security-oriented libraries", given enough time. Everything is a hacking target eventually.

In real world this would just mean that more bugs are for longer unpatched.

In the majority of cases: Absolutely. That is basically the direction things have gone with "just docker build the image and redeploy".

The amount of frustration dynamic linking causes is just not worth it in today's day.

I think that calculus looks very different 20, 10, maybe even 5 years ago, when getting packages rebuilt and distributed from a zillion different vendors was much harder.

This. If we'd just statically link executables, we wouldn't need Docker. Docker is a solution for a problem that only exists because of dynamic linking, and the tool's popularity should be a wakeup call.

This is painfully reductionist. The mount namespace is only one of multiple namespaces (eg. pid, network, user, etc.) that containers utilize. Security doesn't stop at linking and shipping binaries.

That's not true.

Docker/containers do far more than just bundle shared libraries. We use a container for almost all of our CI builds. Our agents are container images with the toolchains, etc installed. It also means it's trivially easy to reproduce the build environment for those occasional wtf moments where someone has an old version of <insert tool here>.

But if the tools were all statically linked, couldn't you ship a zip with the binaries needed for building?

How do you update those tools and make sure you've deleted all the old versions? What do you do to avoid the problem of a developer SSH'ing and apt-get installing a build dependency?

Tools aren't the only part of the environment though. Setting the JAVA_HOME environment variable for a build can dramatically change the output, for example. Path ordering might affect what version of a tool is launched.

They also mean that our developers don't need to run centos 7, they can develop on windows or Linux, and test on the target platform locally from their own machine.

Setting JAVA_HOME matters, again, because the Java runtime environment is essentially dynamically linked to the Java program you’re running.

The nice thing about fully statically linked executables can just be copied between machines and ran. If Java worked this way, the compiler would produce a (large) binary file with no runtime dependancies, and you could just copy that between computers.

Rust and Go don’t give you a choice - they only have this mode of compilation.

Docker forces all software to work this way by hiding the host OS’s filesystem from the program. It also provides an easy to use distribution system for downloading executables. And it ships with a Linux VM on windows and macos.

But none of that stuff is magic. You could just run a VM on macos or windows directly, statically link your Linux binaries and ship them with scp or curl.

The killer feature of docker is that it makes anything into a statically linked executable. You can just do that with most compilers directly.

java home is an example. CC is another one for most C build systems, goroot.

> But none of that stuff is magic.

I never said it was, I simply said that docker does more than statically link binaries.

>You could just run a VM on macos or windows directly, statically link your Linux binaries and ship them with scp or curl.

https://news.ycombinator.com/item?id=9224

> The killer feature of docker is that it makes anything into a statically linked executable.

The killer feature of docker is it gives you an immutable*, versionable environment that handles more than just scp'ing binaries.

> CC is another one for most C build systems, goroot.

The benefit docker provides here is giving users a simple, reproducible, downloadable environment with no external dependencies.

I'm not saying Docker is useless in the current build environment. On the contrary - Having a convenient way to sandbox, ship and execute arbitrary execution environments and statically linked executables is really useful.

I think this is such a great idea we should make debian work closer to this model. Nix is a fantastic step in this direction and it'd be great if more distributions followed suit.

Debian and friends fill your hard drive with of weird, non-reproducible versions of various dynamically linked libraries. The fact I can't take a binary on my x86_64 linux machine and run it on your x86_64 linux machine is an embarrassment. Moving toward static linking and fully reproducible builds would be a tremendous help here.

Docker feels like a kludgy hack. I'd love a world where docker isn't necessary because its best ideas have been lowered into the OS itself.

Indeed, you could. Linus takes backward compatibility more than a little seriously.

Where does static linking come into play? Why can't you ship a zip with binaries and libraries?

It's plenty common for binary distributions of software to come as tgz files. That's usually including a bunch of .so files.

> It's plenty common for binary distributions of software to come as tgz files. That's usually including a bunch of .so files.

Yes, and compatibility across distros / releases is very hit-and-miss, because they usually ship certain libraries and rely on the OS for others.

I’m not sure whether it’s possible to ship every library without running into ODR issues, but at that point, you’d loose all of the advantages of dynamic linking anyway.

No idea, but it has been common since forever, and of course you miss the advantages of dynamic linking (the ability to update libraries separately) which is the whole point.

Docker provides much more isolation than a dynamic libraries.

Static linking is not a new thing. It was there since a beginning. Dynamic linking is a new thing compared to it.

> Docker provides much more isolation

The very opposite. You can do all the sandboxing you want without needing docker and you will avoid docker's big attack surface.

Not without machine code analysis. That's a pretty big attack surface itself if you're relying on it for safety properties.

I don’t disagree, but I think there’s a productive middle ground here.

Docker has cleared become popular because it drastically eases application deployment and cleanup.

Static linking also solves a huge number of these problems, and in a generally better way.

Docker has overhead, an associated attack surface, bulky artifices, etc but it yields very nice benefits for workload isolation (env configs, storage mounts, etc) as well as cross OS/arch builds.

Is there a solution that combines the best of these options? A super-slim docker for statically linked apps that combines the ease of deployment with a much smaller footprint?

> A super-slim docker for statically linked apps that combines the ease of deployment with a much smaller footprint?

I feel like this could be a simple shell script, at least for 99% of real-world use cases.

Not really, you'd still have a problem of building with possibly conflicting system dependencies, packaging (a dev team that can build its own .debs or .rpms is a rarity), ease of deployment / distribution, and running multiple instances. Developers don't want to deal with that crap.

That is so wrong. I use docker to build things which I cannot build on my host system because the host and build have incompatibilities. Static locking would not solve this.

No, both docker, static linking and flatpak share the same issue with being unable to provide reliable and tested security updates.

Once you bundle multiple libraries into blobs you always have the same security nightmare:

https://www.researchgate.net/publication/315468931_A_Study_o...

(and on top of that docker adds a big attack surface)

This argument comes up every time this discussion comes up (on flatpaks etc). And for almost all systems I’d say “yes, absolutely”.

One headache over another.

What’s to guarantee applications haven’t statically linked a library (in whatever version) rather than linked a “system” version of a library?

With the dynamic one you have both problems…

Flatpaks won't be automatically recompiled though. It might take years before they bump whatever vulnerable thing they are using.

> What’s to guarantee applications haven’t statically linked a library (in whatever version) rather than linked a “system” version of a library?

It's easy to see which commands are given to the compiler and if there is a copied library if you have sources.

> Flatpaks won't be automatically recompiled though. It might take years before they bump whatever vulnerable thing they are using.

If we made all builds reproducible (like oasis) we could setup an automated trusted system that does builds verifiably automatically (like what Nix has).

There are already trusted sources of software packages that provide security updates: traditional Linux distributions.

Doing automated, local patching and rebuilding is far from enough. You need experts to correctly backport patches and test them thoroughly.

No, no… We need to move very fast, and break all the things. Tradition and conservatism is not the way. We need to compile everything and anything. Change the paradigm, break the status quo!

Oh, just like Gentoo, Slackware and LFS. ;)

Seriously, maybe we shouldn’t ignore history and learn why things are like this today.

People bashing distributions seem to ignore that the alternative is the google play store.

Which reproducible builds project is striving to do for quite some time.

It’s not just a simple switch you flick and recompile.

Yes but in general a system can safely be assumed to contain At least some software which was built elsewhere. Especially in more proprietary worlds like on Windows and MacOS.

> And for almost all systems I’d say “yes, absolutely”.

Do you even understand that you are advocating in favour of perpetually vulnerable systems?

Let's not touch the fact that there are vague and fantastic claims about shared libraries being this never ending sources of problems (which no one actually experiences) that when asked to elaborate the best answer is wand waving and hypotheticals and non-sequiturs.

With this fact out if the way, what is your plan to update all downstream dependencies when a vulnerability is identified?

Before you decry statically linked systems as "perpetually vulnerable", I'd advise you take a look at the number of privilege escalation vulnerabilities caused by LD_PRELOAD alone.

> Before you decry statically linked systems as "perpetually vulnerable"

But statically linked systems are perpetually vulnerable by design, aren't they?

I mean, what's your plan to fix a vulnerable dependency once a patch is available?

Are you tracking each and every dependency that directly or indirectly consumes the vulnerable library? Are you hoping to have access to their source code, regardless of where they came from, and rebuild all libraries and applications?

Because with shared libraries, all it takes to patch a vulnerable library is to update just the one lib, and all downstream consumers are safe.

What's your solution for this problem? Do you have any at all?

What's your plan if fixing a vulnerability requires changing the size of a struct?

> What's your plan if fixing a vulnerability requires changing the size of a struct?

Do you actually have an answer any of the points I presented you, or are you going to continue desperately trying to put up strawmen?

If any of your magical static lib promises had at bearing in the real world, you wouldn't have such a hard time trying to come up with any technical justification for them.

The argument that you seem to be making is that under dynamic linking, there is never any reason to need to recompile an application, and thus the fact that static linking forces you to recompile applications makes it non-viable from a security perspective.

Unfortunately, this is very far from being true in the real world.

And even if all libraries did maintain 100% ABI compatibility forever, and even if there were never any compiler or processor bugs that needed to be mitigated by recompiling applications, dynamic linking would still add runtime complexity and thus surface area for security vulnerabilities.

> Are you tracking each and every dependency that directly or indirectly consumes the vulnerable library? Are you hoping to have access to their source code, regardless of where they came from, and rebuild all libraries and applications?

Yes? I mean, are you just throwing random untrusted binaries onto your servers, not keeping track of what their dependencies are, and hoping to God that you will never ever need to recompile them? (And if your explanation for that is "I use proprietary software from vendors who refuse to share source code", have you chosen incompetent vendors who cannot respond to security vulnerabilities in a timely fashion?)

What is your plan?

Four-part plan is this:

1. For each application and library in the system, maintain a copy of the source code and the scripts necessary to rebuild it.

2. Keep track of the dependency tree of each application and library in the system.

3. If there's a problem with a dependency, update it and rebuild all dependent applications.

4. If I'm using binaries from a vendor (whether that's a Linux distribution or a proprietary software company), the vendor needs to be responsible for (1) to (3).

In practice, your package manager will generally implement almost all of (1) to (3) for you. (Newfangled systems like NPM or Cargo usually do it using lockfiles and automated tools like Dependabot.)

If you're using dynamic linking, guess what? You still have to do these four things, because even with dynamic linking, there are still problems that can only be solved by recompiling your software. Even if you think reasons like ABI incompatibility or processor bugs aren't compelling, there might be, y'know, bugs in the applications themselves that you've gotta patch.

> If I'm using binaries from a vendor (whether that's a Linux distribution or a proprietary software company), the vendor needs to be responsible for (..)

I'm not even going to bother pointing how unfeasible all your other points are. I'm just going to point this Fack: you do understand this does not work and never worked at any point in time, don't you?

Do you understand the explicit reference to perpetually vulnerable systems? Do you realize where it cames from?

You only have the power to rebuild the packages you own personally, and even so static libs offer zero ways to keep track of which version went into which build.

Once your fantastic panacea starts to rely on your idea to force third parties to follow your personal orders to make new releases under your own personal terms, you should be very aware that you will not get your wish. You'll instead just keep on using the same vulnerability-riddled release.

Do you understand the importance and value of static libs? You do not need four-point authoritarian and deeply impractical and unfeasible plans to keep your system safe. With shared libraries you just patch the one lib, and all your system is safe.

And again what tradeoff do you want to achieve for this? Nothing?

That's like claiming that a leaky sump pump sank the titanic.

Linux has never been anywhere close to resistant to privilege escalation bugs.

Google “OpenSSL dependency conflict” and view the results. There is significantly more work going on when OpenSSL releases and update than “just update the single shared library file.” Applications often need to be updated as well, so you may as well just statically link the new binary.

Haven't seen a single OpenSSL issue over the past 25 years of using Linux.

libstdc++ errors? Seen them.

Libstdc++ errors notwithstanding, what about heartbleed?

https://en.m.wikipedia.org/wiki/Heartbleed

I think the parent was talking about dependency conflicts, not issues in OpenSSL.

It really depends on whether the bug is in API or not. If it is not, it is likely that upgrading only the shared library is enough.

How lovely to live in a world where api contracts are absolute and complicated internal behaviors never conflict in unforeseen ways. We’ve really gotten on top of the software complexity problem

> How lovely to live in a world where api contracts are absolute and complicated internal behaviors never conflict in unforeseen ways.

Some libraries managed to preserve API contracts throughout the years just fine.

Even in C++, a language which does not establish a standard ABI, has frameworks which have not been breaking the ABI throughout multiple releases, such as Qt.

That's exactly the problem deb files and apt solve (via integration testing and documenting antidependences.)

> you'd rather every single binary that has it statically linked is updated, rather than a single shared library file?

Yes! A thousand times yes. I'd rather reinstall every single compromised program than deal with the ridiculous complexity of dynamic linking.

In some cases it’s impossible to build older software, or very difficult.

Considering the complexity that shared libraries add over static linking: yes absolutely.

Is that only that good libraries should correctly use semver (i.e. be backward-compatible unless they change the major version), or is there complexity somewhere else?

My feeling is that dynamic linking is a pain because sometimes one needs, say, 1.2.5 for some part of the code and 1.3.1 for another part, and they are not compatible. Which hints towards the library author having messed up something, right?

We just didn't update Log4j since it broke a critical, old, closed source offline program. Vendor has long dropped support for this old program. Cue that up to the usual executive missteps that we get from up high.

Yay, we get to keep on using old Log4j now because one program holds it back.

If the "old log4j" is 1.2.x, you can update to reload4j (https://reload4j.qos.ch/), which is a fork of the latest log4j 1.2.x by its initial author; you don't have to update to log4j 2.x (which changed the API). And you might not even need to update, since "old log4j" (1.2.x) wasn't affected by that vulnerability in the first place (it has its own old vulnerabilities, but they're all in code which is usually not used by real world deployments).

Yes. I can think of worse problems w shared libs. Should we start a list?

If you keep around the .o / .a files this isn't so bad.

you only need to relink, not recompile. i think oasis caches artifacts so there's no overhead.

Sure. Networks are fast enough.

What about closed-source binaries where a patched binary can't be obtained from the vendor?

With closed-source binaries the argument for static linking is the strongest, IMHO, or at least shipping all the required shared libraries and LD_LIBRARY_PATH that in. The amount of times I've had to muck about to make a five-year old game work on a current system is somewhat bonkers, and I would never expect someone unfamiliar with shared library details (i.e. a normal non-tech person, or even a tech person not familiar with C/Linux) to figure all of that out.

Closed source programs almost universally bundle all dependencies (except system ones) irrespective of whether they are using static or dynamic linking. They have to to ensure that they actually run reliably.

The same thing I do if the vulnerability is in the binary itself, and not in a library.

Surely, if you care, you have a mitigation plan for this?

You could link locally, or you could opt not to use them. Or you could restrict dynamic linking to these special cases.

Binary patch it.

Or you can link locally.

Yes