there are as always tradeoffs, but a few more arguments against dynamic:
by opening up a path (automatic updates) to a third party, you're providing a wide vector for those updates to cause new* problems (correctness, security) for your deployment, as well as fix old ones
transitive api compatibility across these upgrades is more of a statement of faith and effort of will than any real guarentee. in reality the compatibilities between the various components are piecewise, and if the graph gets big enough it can be really hard to find a compatible set of libraries.
runtime binary integration is always strictly worse than build time source integration, since there is tooling in the latter promoting consistency, and basically none in the former
since you'd really like to be able to deploy changes in your own* software with very little effort - dependency updates can ride on the same train for free.
presumably you have to have some intervention to restart your service against the new dependencies regardless. at that point its a question of the relative complexity of a dynamic restart of an existing service vs a build and deploy. in the limit these are both buttons.
from that perspective, dynamic updates are a helpful crutch for people who don't have a good handle on their own build, test and deploy. thats pretty far from being an arguable fundamental good.
Are you familiar with the functional package management approach of tools like Nix (https://nixos.org/nix/) and Guix (https://www.gnu.org/software/guix/)?
They use dynamic linking but store all packages in a content-addressed format, outside the usual FHS. Packages are free to rely on separate versions of their dependencies as needed. Packages are defined in a 'source-based' format, but because of the content-addressable storage strategy, both Nix and Guix can offer transparent binary caching.
Guix folks in particular are working on verifying the integrity of binary caches, so you don't have to blindly trust them, and they've developed a system they call 'grafting' which can safely and quickly replace vulnerable packages without recompiling.
Both Nix and Guix have some container infrastructure as well. I'm not as familiar with the Guix side, but Nix has tooling for deterministically generating Docker images and systemd-nspawn containers, so you can leverage these packaging tools at the same time as existing container deployment infrastructure.
I like your analysis of 'build time' vs 'runtime' integration of dependencies, and I think you might enjoy playing with/critiquing Nix and Guix, because they address these problems in a strange and powerful way. They use dynamic linking for deduplication and small upgrades, but try very hard to integrate everything they can at build time. I'd be interested to see where you think their approach is strong and weak, if you ever get a chance to try it out.
What makes Nix more interesting than just that is that it actually provides tools to manage the dependencies and build. As much as I think owning your whole world is important, it can really consume alot of engineering resource just due to the state of the world. Since Nix signatures are transitive (as I understand it) it also captures the versions of the included tools, which is usually too far for most organizations to go. And while issues having to do with compiler versions are rare, they can really consume alot of energy trying to track down.
So, without having any firsthand experience, it seem like its as dependable as static linkage, and may be quite a bit more convenient.
There's a certain positive aspect to the treadmill effect you get with operating system upgrades that gets blunted once you can run another version of everything in a container nobody has rebuilt in ages.
But in that case you might have 3 teams putting pressure on the Dingus team to upgrade their stuff. If they can just retort with "run it in a container dummy" then they have no incentive to ever fix it.
No. Distributions have been ensuring security for 20+ years by:
- using dynamic linking
- unbundling/unvendoring dependencies into dedicated packages
- having a security team that updates vulnerable packages in a timely manner
- having tools to update vulnerable packages without having to rebuild/reinstall/reconfigure the world
This is what happens when you don't do that:
https://blog.acolyer.org/2017/04/03/a-study-of-security-vuln...
That is true, except in the case where you aren't empowered to do the building. Then dynamic update is better than no update.
This is rather harsh on "plugin" architectures, COM, and OS libraries.
I seriously don't get how folks are comfortable with using all these binary layers built and rebuilt from unspecified versions of other binary layers, all shipped with no cryptographic verification by default. (Transport security with SSL is good enough, yeah? sob)
It's thankfully a very interesting problem to have to address, but it does mean that our time-to-market is inherently atypical compared to others.
This class of problems has been around for ages, the noise is just getting louder every day.
Double down on the Dockerfile format and abstract things that would otherwise be frozen in time, taking ownership of some aspects of DockerHub that are currently managed by contributors. They do this already with official language images but there has to be more to it outside of that than expecting people to figure out the intimate details of each base image while keeping them up to date across OS versions and library versions.
Seems like the only path forward if security is a priority is to build the base layers from scratch...
If it's a static library, see my zlib comment. Those have the same problem.
And restart every effected service. You can't "just" run the apt update; the binaries using any shared libraries must be restarted so as to load the newer version of the library. I find this is often the harder problem to solve. (If you can just reboot the machine, do that, it's easier; otherwise you have to track down who was using the old binary and restart each one of them…)
(sorry, this is a rather critical step that I've seen missed before.)
Of course it is arguable that if your system is setup so that you can not afford to reboot a single server, then you are in pretty poor shape to begin with. When I was young, long uptimes were such cool thing. Nowdays I see them more as a liability or smell that something is iffy.
thus making the "positive" side of shared libraries basically look very bad on paper. I think that a lot of systems are affected by that problem. I like shared libraries for a desktop os and applications where you often reboot. But for a server? Hell it's easier to upgrade golang applications/java applications (if the jvm has no bugs and since j9 even bundled) is "saner" to upgrade.
also what a lot of people just don't get is that a sane system mostly only has one service per os/container. thus docker pull vs apt update is basically the same.
No, that's not how it works, at least on Unix systems. Shared libraries are mapped into each process's address space separately; there's no system-wide mechanism that prevents different processes from running different versions of the same library at the same time.
Note that often the actual backing files are disjunct anyway and only symlinks get replaced, due to versioned .so names (as in libfoo.so.2.4.5, libfoo.so.2.4@, libfoo.so.2@, libfoo.so@, you've all seen it).
But doing this has the same problem. You've just re-implemented DLL hell as container hell.
I absolutely agree. So how do we fix this?
Well, first we need to identify what's broken, and then we can talk about how to fix it. We could look at container images and see that, in the pursuit of reproducibility/determinism, they make it infeasible to make critical updates; from this we could infer that determinism is the root of our security woes, and that we have to choose between having a run-time environment that you can depend on, or being able to promptly make critical security updates.
Fortunately, that a false dichotomy. Actually, it's even better than that: it turns out there's a third option that gives you all of the above:
1. The ability to quickly make security updates
2. A deterministic and serializable run-time environment
3. A deterministic and serializable build-time environment
Note the addition of bullet point #3. In something like Docker, you can successfully build an image from a Dockerfile today, and yet it could fail to build tomorrow. You can build a Dockerfile on your machine, but it might not build successfully on my machine. What I am claiming is that there is another approach to containers that would additionally provide determinism when you're building the container's root file system, whereas e.g. Docker only gives you determinism in the sense that a previously built image can be passed around.
So what system could provide all of the above? How could we make that work?
(If you get bored with the details, feel free to skip down to the spoiler after TL;DR)
We'd first need a deterministic build mechanism; broadly speaking, one in which we could build e.g. executables, fetch tarballs, construct file-systems, etc. To be deterministic, we our build system must have certain properties:
* "normal" builds must be cut off from network access, whereas
* A special "fixed output" build type could e.g. fetch tarballs, provided that e.g. a SHA-512 hash is given beforehand
* Each build specification must specify what other build specs it depends on. Each build environment is constructed such that processes therein are only permitted to see/access files that are the resulting artifacts of those explicitly listed dependencies. Arbitrary read/write access to the host file-system is not permitted.
* The resulting build artifacts go in their own immutable prefix, something like, let's say, /store/<hash>-<name>, where <hash> is a unique key derived by hashing the build specification, and <name> is a friendly name for the build artifact. To provide a concrete example, if you were to build `gcc`, you'd have something like /store/4r5kszyy0iirc5agfah45lvz7mnnsrb4-gcc-7.3.0/{bin,include,lib,lib64,libexec,share}, which would all contain the usual suspects. Note that when we hash the build specification, that spec refers to its dependencies, which in turn have their own unique hash, and so on. As a result of this design decision, we have: support for multiple versions of the same software; updates can't break existing executables (due to, let's say, breaking ABI changes in openssl or libc or whatever); and a final point that's important enough to split out as the following bullet point:
* We can trivially construct deterministic environments: write a build spec that takes as an input all of the dependencies you wish to have in the environment, and (as the single resulting build artifact) produce a symlink tree that is the union of all the build artifacts of those aforementioned inputs. For example, you might want an environment with, I don't know, let's say gcc, python and ruby; you would end up with something like /store/m6n3y1rlz9rf06rj4spmvlkmv51angbm-container-env, which would contain bin/gcc, bin/python, bin/ruby, etc. Of course, having this environment wouldn't be much good unless you could actually run something inside of it, so that leads to the next point:
* We can take an environment (as described in the previous point) and use it as the file system for a container: just do all of the usual container/namespace stuff, and e.g. symlink the env to /usr. A byproduct of this is that rollback/forward is easy: just bump a single symlink to the appropriate env path.
* Due to our design, it's a trivial matter to serialize an environment and (re)distribute it.
* Also important: the host system can build and retain all of the container environments in the host's /store. We could then bind-mount paths into the container as necessary. As a result, when two containers depend on precisely the same software, you only pay the price of a single copy. This is unlike Docker in that Docker requires that your image's layers directly inherit from each other; if you have, say for argument, an Ubuntu based image and a CentOS based image, it doesn't matter if there are components that are exactly the same in both - you have to pay the price of two copies.
TL;DR -- TL;DR -- TL;DR --TL;DR
We now have a system in which we can construct recipes to build container environments, and those recipes can be deterministically built, serialized, distributed, and then deserialized and used as a reproducible environment. Thanks to this, it's easy to tweak an existing recipe to bump up, say, openssl when a critical CVE is announced. We've beaten the typical image-based approach to containers (e.g. as used by Docker) in terms of security risk mitigation, disk foot print, developer time (from otherwise fighting with broken build specs / Dockerfiles), etc.
Okay, so we don't actually have this system, we've just fantasized about one...
...unless something like this already exists!
Indeed, this is precisely what you'd get by using the Nix package manager (and you get extra goodness on top if you use the NixOS distribution): you write what amounts to a Dockerfile in a slightly different syntax, but then you get all of the above benefits.
If you've already bought into Docker, you can actually use Nix to construct your Docker images: https://nixos.org/nixpkgs/manual/#sec-pkgs-dockerTools
Some benefits:
* Nix provides reproducible docker images; Dockerfiles don't.
* Faster builds: Nix is better at (safely) caching the results of intermediate steps in your build, whereas a Dockerfile has to be conservative (if you so much as change the whitespace in a RUN command, that step (and all subsequent steps) have to be re-run).
Previous discussion: https://news.ycombinator.com/item?id=16670337
It’s the same lesson that a lot of people learned about static linking, with the zlib vulnerabilities years ago.
https://www.linux.com/news/linux-kernel-netscape-affected-ma...