As an intern at a data science company, I noticed that a certain closed-source shared library we were using was calling the same trigonometric functions with the same arguments over and again. So, I tried to use LD_PRELOAD to "monkey-patch" libm with caching and that reduced the running time of the data pipeline by 90%.
>I never want to see a dynamic library again.
And replace it with what? Completely statically linked libraries? How much disk space would linux userland require if every binary was to be statically linked?
Telling maintainers to dynamically link just in case someone wants to change the libraries is like having replaceable CPUs on motherboards. Useful for 0.1%, but requires a lot of working around.
Of course, there are many independent arguments for using dynamically linked libraries, which is why most linux distributions use them.
What do you mean by this? You can generally replace the CPU on a desktop or laptop motherboard.
This is a very edgy edge case, though.
> How much disk space would linux userland require if every binary was to be statically linked?
Megabytes more?
I have never been on a project or a machine since the 70s where executables took a majority of the disk space. On this disk, about 15% of it is executables, and I have a lot of crap in my executable directory and just deleted a lot of junk data.
Also, no one's seriously proposing that core libraries that everyone uses be statically linked.
Megabytes more in total, or megabytes more per binary?
> On this disk, about 15% of it is executables, and I have a lot of crap in my executable directory and just deleted a lot of junk data.
Is there any study on how this might change if we moved from dlls to statically linking everything?
> no one's seriously proposing that core libraries that everyone uses be statically linked
If we agree that shared libraries are fine for the common case, and there are n number of existing solutions for distributing applications with tricky dependencies, what are we even talking about in this thread? :\
Drew Devault did an informal one https://drewdevault.com/dynlib.html
TL;DR - not at all substantially.
We’re moving to a world where every program is containerized or sandboxed. There is no more real “sharing” of libraries, everything gets “statically linked” in a Docker image anyway.
You could volume mount /usr and chunks of /var to help, but I'm not sure that would be sane.
Imagine you have 10 programs statically linked against the same library. How many will be promptly updated? How do you, as a system admin, track which ones were not?
In production you very often end up maintaining stuff not from distribution, you have to track and rebuild that anyway.
Sure, conceptually there may be a better solution to both problems, but it has not been deployed by any GNU/Linux distribution I know.
And I am not sure if dockerizing or building everything statically and hoping a solution will eventually show up one day is the path forward.
At least when the distro updates its shared libraries it fixes it for all users, imagine how many rebuilds + time + effort it would take for every docker container that had the same shitty small bug.
Linking to a library is not what I would describe as "complex", or have any meaningful challenge regarding "interdependence".
This issue makes no sense in Windows, which you are free to drop any DLL in the project folder, and for different reasons also does not make sense in UNIX-like distribution which already provide official maintained packages.
I'm sure that there are plenty of problems with distributing packages reliably, but dynamic libs ain't one of it.
Why? You can have both e.g. /lib/libfoobar-1.0.4.so /lib/libfoobar-1.0.3.so
Usually you would have /lib/libfoobar-1.so -> /lib/libfoobar-1.0.4.so, but it doesn't prevent you from linking the problematic program directly with libfoobar-1.0.3
I feel that patchelf[0] is an ingenious little tool for exactly that purpose that isn't getting enough love. Not as useful for FOSS stuff, but it's been really useful for the times I had to relink proprietary things on our cluster in order to patch a security vulnerability or two.
That's incidental and not by principle, isn't it? You should be able to have a multiple versions of a library installed, and the dynamic linker would pick the appropriate version.
Heck you could even have multiple versions in the same process (if you depend on A and B, and B uses an older version of A internally). Unfortunately I think the current dynamic linker on Linux has a shared namespace for symbols.
I also disagree that it is not useful to be able to update libraries separately. The example that I like to give is a PyGtk app written against an earlier Gtk2. Without modifications, it ran against the very last Gtk2, but now supporting: Newest themes, draggable menubars, better IME, window size grippers, and all kinds of other improvements to Gtk.
In the days of ActiveX (and I realize this is a bad example because in reality you often had breakage), you could update a control and suddenly your app would have new functionality (like being able to read newer file format versions or more options to sort your grids and so on).
I've not experienced that. If the package is maintained at all, then it will get updated. People are still somehow maintaining Perl 5 libraries and putting out bugfix releases, which are functionally equivalent to dynamic linking.
If it's not maintained, then of course it will get removed from the distribution, for that and for a number of other reasons.
Not to be pedantic, but having multiple versions of any Perl library, or multiple versions of Perl on a single server without things getting stepped on is trivially easy. There's utils to switch Perl versions as well. I mean, it's Perl.
Perl has a long, long, long history and culture of testing, backwards compatibility, Kwalitee Assurance, and porting to every system imaginable behind it as well, so if some Perl script from the 90's still runs with little to no modification, that should not exactly be seen as a rarity.
There are a lot of reasons to prefer static linking, but to me the argument of "it means you don't need to keep your package up to date with bug and security fixes" never held up, from a distro point of view anyway.
Then what is? Dropping a custom library next to a binary tends to be way easier than modifying the binary.
The distributions will take all front-line bug reports and the maintainer will forward them to upstream only if applicable. Or at least, that's how Debian operates.
It really isn't. At all. For example, let's take a notoriously egregious example of C++, which doesn't really have a stable ABI, and a major framework like Qt, which provides pretty much the whole platform. Since the introduction of Qt5, you can pretty much upgrade even between LTS releases without risking any problem: replace libs built with the same compiler tool chain, and you're ddone
This is not an isolated case. Shared libraries and semantic versioning is not a major technical hurdle, and one which in practice only has upsides.
as someone who develops Qt apps for a living, this is not true at all. Yes, Qt promises no ABI / API break, sure. But it doesn't matter at all when a new release introduces a bug that breaks my app. It's super common to have to pin software to Qt versions if one wants to minimize bugs (and more importantly have a precise behaviour for all the users of the software - for instance, the way hidpi has been implemented has changed something like 3 times across the Qt5 lifetime, and there are still things to fix ; this means that on a hidpi screen the software won't look the same if linked against qt 5.6 / 5.10 / 5.15 which is pretty bad as for instance assets may have been made specifically to fit with the "Qt 5.6" look and may appear broken / incorrectly positioned / etc... on later versions (and conversely).
I fail to see the relevance of your case. Semantic versioning and shared libraries is a solution to the problem of shipping bugfixes by updating a single component that's deployed system-wide. Semantic versioning and shared libraries do not address the problem of a developer shipping buggy code. Shared libraries ensure that you are able to fix the bug by updating a single library, even without having to wait for the package or distro maintainers to notice the bug exists.
> It's super common to have to pin software to Qt versions if one wants to minimize bugs (and more importantly have a precise behaviour for all the users of the software - for instance, the way hidpi has been implemented has changed something like 3 times across the Qt5 lifetime, and there are still things to fix ;
I've worked on a long-lived project based on a Qt application that supported on all platforms, and we've upgraded seamlessly from Qt 5.4 up to 5.12 without touching a single line of code.
In fact, the only discussions we had to have regarding rebuilding the app was due to a forced update to Visual Studio, which affected only Windows.
YMMV of course, specially if no attention is paid to the contract specified by the library/framework, but I disagree that your personal anecdote is representative of the reality of using shared libraries, even in languages which are notoriously challenging to work with such as C++.
but that assumes that the only thing that changes between library_v1.0.0.so and library_v1.0.1.so is the bug. There are always tons of regressions and unrelated changes between minor Qt versions - hell, I even had patch versions break, for instance I was hit by this which was introduced in 5.11.1: https://bugreports.qt.io/browse/QTBUG-70672 or this which worked before 5.9 and failed afterwards (for good reasons) : https://bugreports.qt.io/browse/QTBUG-60804 or this in 5.4: https://bugreports.qt.io/browse/QTBUG-44316
> and we've upgraded seamlessly from Qt 5.4 up to 5.12 without touching a single line of code.
and that worked on mac, windows, linux with X11 & wayland, in low-dpi and high-dpi alike ? I don't see how if you care about your app's looks (e.g. your UI designers send you precise positioning & scale of every widget, text, image, etc).
hell, if you have a QtQuick app today, resizing its window has been broken since Qt 5.2 (since then, layouting happens asynchronously which gives the impression the app wobbles - and that is the case for every Qt Quick app under Linux, even the simplest hello world examples ; this worked much better in 5.0 / 5.1, see e.g. https://bugreports.qt.io/browse/QTBUG-46074 ).
It's not true at all. Libfoobar-1.0.4 is backward compatible with libfoobar-1.0.3, so both programs will work. If one of the programs or the lib has a bug, then it's easier to just fix the bug than to add burden on maintainers.
Moreover, both libfoobar.so.1.0.3 and libfoobar.so.1.0.4 can be installed at the same time, e.g. libfoobar.so.1.0.4 can be packaged as libfoobar-1.0.4, while libfoobar.so.1.0.3 can be packaged as compat-libfoobar-1.0.3.
If you have more questions, try to find answers in packaging guidelines for your distribution, or ask maintainers for help with packaging of your app.
It's just not worth it. A binary is a closure over the entire source text of the program and its libraries. That's what CI tested, that's what the authors developed against, and that's what you should run. Randomly changing stuff because you think it's cool is just going to introduce bugs that nobody else on Earth can reproduce. Nobody does it because it absolutely never works. You hear about LD_PRELOAD, write some different implementation of printf that tacks on "OH COOL WOW" to every statement, and then never touch it again. Finally, dynamic loading isn't even the right surface for messing with the behavior of existing binaries. It has the notable limitation of not being able to change the behavior of the program itself; you can only change the results of library calls.
I never want to see a dynamic library again. They have made people's lives miserable for decades.