It's not for disk space, it's runtime RAM occupation usage. The code for a .so is going to be slightly larger than the equivalent .a. At the very least, you'll need to burn extra space for the PLT and GOT. And PIC relocations are going to tend to burn more space than static linking relocations.
The advantage of a .so you're talking about is that you can load a library shared by multiple processes at a fixed virtual address location that's the same for all processes, and thus you only burn the library once in RAM. For static libraries, the embedded addresses are going to be slightly different depending on their corresponding libraries, so even magic deduplication isn't going to be able to keep you from having the library in multiple places in physical memory.
Ancient Linux distributions once had something called "a.out shared libs". These were mapped to fixed addresses. The distribution had to work out non-conflicting addresses for all available libraries. The build procedure for a.out shared libs involved intercepting the assembly output of GCC and doing some text processing on it.
During the (static) linking of libraries, an extra metadata item would be stored in the ELF file: the canonical pathnames of all the static libraries used, along with base addresses of R/X segments (code) and R/O segments (r/o data). Upon forking a process, the information would be provided to kernel, and passed over to the KSM, to use as basis of same-page lookup and possible merging of the pages.
This effectively inverts the current mechanism. Currently when libraries are dynamically loaded, the dynloader uses actual file pathname of the shlib as the key for operation opposite to "merging" - i.e., re-uses the already loaded R/X and R/O pages, by adding proper memory mapping.
The proposed change is three-fold:
1) extend the linker to add the metadata to ELF,
2) extend in-kernel ELF interpreter to extract the info upon exec() and friends,
3a) extend the KSM with a limited mode, where it would look up & merge only the hinted memory regions, in linear fashion, right after an exec() & friends.
An alternative to 3a), to avoid fussing about with KSM: 3b) modify the VM subsystem to extend current swap/SHM so it provides an unique address range for each static lib canonical pathname. Requesting pages from this address range would map from the shared pages, if any process already loaded one such.
To handle adversarial processes on single machine, a further extension where crypto signatures are checked is possible.Also, it's quite likely that static linked libraries wouldn't be located at the exact same page offsets every time they're linked in to a binary, making page-level dedupe not useful.
Linux does have something called "KSM" that de-dupes memory pages, but it appears that it's only so a hypervisor can de-dupe guest OS pages? Odd that it's not more general than that. Though I guess there is a cost: the KSM code needs to scan through memory to find duplicate pages, which I can't imagine is fast.
Shared libraries are a goofy hack dating back to a time when people mistakenly thought they needed them, and they need to go away.
Err, we're running Electron apps today, with a full blown embedded browser engine, a vm, and a hellish complex DOM for rendering. Others run apps inside containers, with their own basic OS and standard libs -- compared to that the above is negligible.
Some of us try not to on the belief that they don't perform acceptably.
The real problems with static linking don't have to do with fork, except in the sense of increasing reliance on VM overcommit (which IMO is a bad idea already). It's more to do with masking reuse of the same library across unrelated processes, as an efficiency issue but even more importantly in terms of tracking dependencies and updating software. No matter how good your configuration management (or similar mechanism) is, rebuilding large-N statically linked executables is less efficient and more error-prone than using the same information to update small-N shared libraries.
I don't think we need to get into my optimization street cred, but suffice it to say I've never used Electron and couldn't really tell you exactly what it is.
Electron is an embedded web-browser meets cross-platform application platform.. it's bloated, slow and awful... it makes Java look snappy, lightweight and brilliant.
You mean for your code, deployed on your systems. Yeah, that works fine.
Now ship a binary to a client system with your static TLS implementation, JPEG decoder, LZ decompressor, whatever. And then go hire someone to watch the CVE feeds every day to know when you have to get them to install an upgrade.
Of, if you don't want it to be your fault when they get pwned, you could just link against the system libraries and tell them to stay updated.
"Static only" makes sense for web apps developed and deployed within the same organization, for embedded solutions where the whole update/upgrade process is known to be under the control of the developer, and... basically nowhere else.
This advice is bad, sorry.
Things have in fact gotten much better over the past years, though. Largely because developers (even of games) have moved away from this architecture and onto managed runtimes like Java and .NET where the runtime and system provide cleaner separation of dependencies and don't require the "static all the time" nonsense that was the norm since the beginning of PCs.
The problem described in the article appears when you dynamically load/unload shared libraries. This only comes up in the context of plugin systems, or sometimes hot code reloading, and in those cases there is no alternative; there is nothing you can do with static binaries to emulate this behaviour.
* Speaking in my own capacity
First, there’s load order which can mean that you’re not actually loading the DLL you think you are.
The second is WOW64 (Windows on Windows). Rather than using fat binaries like Linux and Mac OS, Microsoft decided to use a “brilliant” strategy where the file system lies to you in the “right” ways so that old 32-bit folders are moved to new addresses and the old 32-bit named golfers hold 64-bit binaries. This means that unless you use the “I really mean it” escape mechanisms (e.g. the “sysnative” directory) 32-bit binaries on 64-bit windows is actually loading DLLs from different paths than the code says.
A primary goal that you don't really cover is conserving not just disk space, but memory. The disk space saved is a proxy for memory savings, which are (even more) valuable. Shared libraries linked in to multiple independent binaries can consume little to no additional memory due to the magic of virtual memory. The same does not hold true for static-linked libraries in multiple binaries.
The reason I ask is, even as "bloated" as software is these days, I could fit ~100,000 fat 10 megabyte Go binaries on my SSD, without bothering to reach for compression. In practice, binaries on disk always pale in comparison to the amount of disk space I spend on media files, caches, backups and imaging, etc.
Perhaps your concerns about virtual memory are true, but looking at top on a machine running Firefox with many tabs open, Discord (an Electron based application,) 3 GNOME Terminal windows, HexChat, Wine, and Telegram... I still don't crack 100 tasks. Still.*
How about a server environment? Well, saving memory is certainly valuable on a server, but I would fail to imagine a scenario where you want so many different distinct binaries running that it would make a major difference in utilization. The shared memory savings that come from dynamically linked libraries are effectively removed when using Docker because each Docker container is going to have its own isolated system libraries and many of them probably won't be the same anyways, and yet I've never had any Kubernetes system where the majority of memory usage came from binary sizes.
I suppose if every single system binary was statically linked, you would maybe be able to notice some kind of difference. But honestly, probably not, unless you were really searching. Usually disk usage of binaries is a much less important concern than CPU and RAM usage, especially across large fleets.
*Though, that was only checking under my local user; it does turn out that there are a fair bit more tasks, but a good amount of them are actually kernel tasks. There's 400 counting kernel tasks, but only 191 not counting kernel tasks. Still not a very large number, in my opinion.
In name of stability, one may say to run external processes with shared memory instead, but now imagine the resource usage of something like InteliJ or Eclipse using only processes for their plugins.
And on an 8GB machine, which most business still use nowadays.
This kind of desktop software, each feature can be its own plugin, meaning menu entries, UI widgets, integration with external tool, database drivers, ….
Now imagine that to the extreme that every class could be a possible plugin, and even 300 will be a low bar.
IDEs are only one exemple, there are plenty of use cases on desktop software, like music, graphics editing for two other common cases.
Browsers are VM managers. Their tabs are the tabs of the VM manager like a VirtManager for KVM or VmWare vSphere. Browsers boot a JS VM fast over HTTP and then communicate over the same HTTP.
Not on iOS?
More or less the same happens with C++ header-only libraries, which are basically uncompiled static libraries. They are very nice and possibly allow strong optimization, but are rather painful to handle in a distribution.
> Plus with a static binary, if you really care about order of loading/unloading etc (which you ideally shouldn't) its trivial to manage with a custom linker script.
Why? It seems to me rather sensible to deinitialize resources in the opposite order in which you initialized them. This way the time span of different resources are contained each within the other instead of just overlap. When using RAII in C++ (which is basically just atexit on a finer grain) it often makes sense to have this requirement.
Without dynamic linking you duplicate all that stuff and, even worse, cannot be certain that all your executables use the same version.
The first reason was disk space: object files could simply be smaller. Nowadays, for all intents and purposes, disk space is free and unlimited.
The second was the ability to ship fixes to system facilities. These were fragile, leading to even more complex methods such as versioning, so though this was a good idea in principle, in practice it was very painful. I believe the Windows folks had their own name for it, "DLL Hell". The npm people still suffer from this from time to time, sometimes notoriously.
As for the fixes: software isn't updated on magtape any more, and most systems have robust upgrade systems. In addition there are all manner of quasi-hermetic isolation systems (VMs, docker images, venvs and the like) so why not just ship a static binary?
Plus with a static binary, if you really care about order of loading/unloading etc (which you ideally shouldn't) its trivial to manage with a custom linker script.
My only exception to this is the kernel: it can be upgraded, and mostly promises not to vary system call semantics too much, so it's OK to me not to link the kernel into my binary :-).