If you require a library, you can specify it as a dependency in your dpkg/pacman/portage/whatever manifest and the system should take care of making it available. You shouldn't need to write custom scripts that trawl around for the library. Another approach could be to give your users a "make install" that sticks the libraries somewhere in /opt and adds it as the lowest priority ld_library_path as a last resort, maybe?
This was the biggest pain point in deploying *application software* on Linux though. Distributions with different release cycles providing different versions of various libraries and expect your program to work with all of those combinations. The Big famous libraries like Qt , gtk might follow proper versioning but the smaller libraries from distro packages - guarantee. Half of them don't even use semantic versioning.
Imagine distros swapping out the libraries you've actually tested out your code with with their libraries for "security fixes" or whatever the reason. That causes more problems than it fixes.
Custom start up script was to find the same xml library I've used in the tar ball i packaged the application in. They could then extract that tar ball wherever they need - including /opt and run the script to start my application and it ran as it should. Iirc we used to even use rpath for this.
This is a red herring. Distros existed before semantic versioning was defined and had to deal with those issues for ages. When packaging, you check for the behaviour changes in the package and its dependencies. The version numbers are a tiny indicator, but mostly meaningless.
To me it seems more attractive than how Nix does it, but I guess they considered such and saw conflicts, therefore went with hashes.
Recently in the python ecosystem the `uv` package manager let's you install a package as it was on a certain date. Additionally, you can "freeze" your dependencies to a certain date in pyproject.toml. So when someone clones it and installs dependencies it will install it at the date you've chosen to freeze it at.
Personally I love this method much more than versioning.
I think versioning is mostly useful to just talk about software and maybe major major additions/changes e.g io-uring shipping with linux mainline.
Yeah, but the package maintainer for a widely used library doesn't actually have the resources to do this. (heck, a package maintainer for a non-trivial application likely doesn't have the resources to do this). Basically they update and hope to get some bug reports from users.
So it is complicated and there is no solution for every context, therefore we use best approximation.
I don't believe that it causes more problems than it fixes. It's just that you didn't notice the problems being silently fixed!
There are issues related to different distros packaging different versions of libraries. But that's just an issue with trying to support different distros and/or their updates. There are tradeoffs with everything. Dynamic linking is more appropriate for things that are part of a distro, because it creates less turnover of packages when things update.
Now a lot of stuff is contnuously buggified so there is no concept of stable and in-progress.
It depends a lot on ABI/API stability and actual modularity of ... components. There's not always a guarantee of that.
Shared libraries add a lot of complexity to a system for the assumption that people can actually build modular code well in any language that can create a shared library. Sometimes you have to recompile because, while a #define might still exist, its value may have changed between versions, and chaos can ensue - including new, unexpected bugs - for free!
Static linking has its place pace, no doubt, but it should not be the norm.
The former often makes a lot of sense in terms of deployment or maintenance - it's all really 1 big system so why rebuild and deploy 60 parts to change 1 when it's all done by you anyways. I'm not sure that's really what every build environment should default to assuming is the use case though but going shared libraries makes a ton of sense for that regardless.
60 programs maintained and built by 3rd parties which just happen to share major version of a library (other than something like stdlib obviously) would seem nuts to manage in a single runtime environment (regardless of static or shared) though!
That, 100% is our scenario.
At the end of the day the apps were a simple end user applications. They used a handful of library functions from different libraries. My users cared about just using my apps to do whatever the apps did. I just care that my app should work on their machines easily - no matter what version of what distro they're using.
Windows programs generally do link dynamically to core Windows libraries—which users are never expected to mess with anyway—and the C and C++ runtimes, but even these can be statically linked against with `cl.exe /MT`. Some programs even distribute newer versions of the C/C++ runtimes; that's where the famous Visual C++ Redistributables come from.
I agree, though—static linkage should be the default for end-user programs. I long for a time when Linux gets 'libc' redistributables and I can compile for an old-hat CentOS 6 distribution on a modern, updated, rolling-release Arch Linux without faffing with a Docker container.
The Linux linking model is so bad. So extremely very bad. Build systems should never rely on whatever garbage happens to be around locally. The glibc devs should be ashamed.
Specifically, what is stopping libc redistributables from being possible? And why disk size limitations from back then caused this situation.
Every time I tried to get a third-party binary app running on Linux, I discovered what the vendor did is they shipped half their dependencies as blobs, and relying on the system for the other half - which is an incredibly brittle system that breaks constantly.
The entry point usually is a script that sets LD_LIBRARY_PATH and then calls into the executable.
This is not true - you can control the DLL path via manifests. There's also a "known DLLs" list in registry which can globally redirect basically any DLL system-wide.
For instance, GNU Make uses some of GNULib.
https://git.savannah.gnu.org/cgit/make.git (gl subdirectory)
LD_LIBRARY_PATH is a debugging and software engineering tool, and shouldn't ever be part of shipped software.
Build systems that make it a huge paint to calculate elaborate rpath, e.g. knowing what the sysroot will translate to on the actual target system or whatever, are just doing counterproductive nonsense.
By the way, I seem to recall that Yocto uses chrpath in order to get its own tools to reference its own copy of glibc that it provides (for itself, in the build environment).
It's simply that when you do want external but bundled neighboring libs, there is a good way to do it.
Suppose whatever you are shipping consists of five executables, which have a few hundred kilobytes of their own code and a 5 megabyte library. If they statically absorb the library, they all become 5.5 megabyte binaries.
Adjust that to whatever sizes you would consider relevant.
You can make it clear when a bug is fixed in a library or one of the executables. If you have a patch update, users can see that only a shared library changed or only a certain executable.
When hunting a bug, you may have certain possibilities (within limits that depend on your library API maintenance), like trying an older executable with the same library, or older library with the same executable.
The program still looks for libc in the system directory.
The argument for this is that if you ship software that features multiple executables that share code, they can have it as shared libraries, and easily find it relative to their own location, just like on Windows.
For example, in places where the a filesystem-related "standard" has changed, I have old static binaries that fail to start entirely, whereas the same-dated dynamic libraries just need to bother to actually install dependencies.
I am convinced that every argument in favor of static linking is because they don't know how to package-manager.
Which would be a fair reason. People who like to build things might just not want to also learn how to package stuff.
If you're linking to libX11 or libgtk or something else that's common, rely on the distro providing it.
I really don't get all the anti-shared-library sentiment. I've been using and developing software for Linux for a good 25 years now, and yes, I've certainly had issues with library version mismatches, but a) they were never all that difficult to solve, and b) I think the last time I had an issue was more than a decade ago.
While I think my experience is probably fairly typical, I won't deny that others can have worse experiences than I do. But I'm still not convinced statically linking everything (or even a hybrid approach where static linking is more common) would be an overall improvement.
If you want a newer version, too bad - your OS doesn't ship that so better luck in the next release. OR you can set up a private repo, and either ship a binary that has the dependencies included (shipping half the userland with your audio player), or they package the newer version of library, which will unwittingly break half your system, if not today, then surely at the next distro upgrade.
It speaks volumes of Linux package management woes, that no vendor ships anything analogous to brew or chocolatey.
What is the gap between e.g. `apt` and Homebrew?
- Packages are useful units of software in homebrew, and both end-user apps dependency libraries in apt (and weird stuff, like headers, or dependencies)
- Homebrew packages are installed for the user, apt is system-wide
- Homebrew packages are updated and maintained usually by people involved with the software they are shipping, apt packages are usually maintained by the distro
- As a result, Homebrew packages almost always work and are almost always up to date, apt ones are almost never (stuff like go, node etc.)
Apt is like a weird frankenstein monster of npm, the system update, and an app store, and all that with a global scope.
It is also a statically linked Linux distribution. But it's core idea is reproducible nix-style builds (including installing as many different versions/build configurations of any package), but with less pl fuff (no fancy funcional language - just some ugly jinja2/shell style build descriptions; which in practice work amazingly well, because underlying package/dependency model is very solid - https://stal-ix.github.io/IX.html).
It is very opionated (just see this - https://stal-ix.github.io/STALIX.html), and a bit rough, but I was able to run it in VMs sucessfully. It would be amazing if it stabilizes one day.
Consider how dynamic linking libc works when a critical security bug is found and fixed. To update your system you update libc.so.
If it were statically linked, you need to update your whole distribution.
Nix is much closer to a "good" dynamic linking solution IMO except that it makes it overly difficult to swap things out at runtime. I appreciate that the default is reproducible and guaranteed to correspond to the hash but sometimes I want to override that at runtime for various reasons. (It's possible this has changed since I last played with that tooling. It's been awhile.)
Depends on what you are trying to achieve though.
Suppose your dev shell contains app Foo v1 which uses lib Bar v2. It links directly against the full nix store path. So loading a different version of lib Bar (say v3) into your dev shell, which puts it on your PATH, doesn't affect Foo - the Foo executable will still link against Bar v2. That's by design and a very good thing! It assures reproducibility and fixes the version incompatibility issue that dynamic linking typically suffers from.
However, what if I want to swap it out for whatever reason? Right now I have to modify the package definition and rebuild it. That's needlessly slow and resource intensive for some one off monkey patch. It also might not be feasible at all on my hardware - most people's rigs aren't cut out to build chromium or llvm, for example.
The UX is also absolutely terrible. Have you ever tried to patch a nixpkgs provided definition? I actually gave up on using nix as my primary build tool over this.
My expectation going in was a neat and tidy collection of independent, standalone package definitions that imported one another to satisfy dependencies as necessary. The reality was a massive monolith containing all packages at once making extracting just one to patch it rather complicated in some cases.
I believe flakes were supposed to address this exact dependency modification UX issue somewhat, but they caused my code to break in other ways and I decided I was done sinking time into tinkering with tooling rather than building software.
> AppImages
AppImages require a large amount of (obsolete) dependencies to run, making their portabiltiy practically worthless. Newer immutable distros like Aeon don't ship the necessary packages to run an AppImage.
1. You can bake the LD_LIBRARY_PATH into your executable with the -rpath mechanism.
2. Both rpath and LD_LIBRARY_PATH expand certain tokens such as $ORIGIN. $ORIGIN expands to the location of the program.
So you can (for instance) set a rpath of $ORIGIN in your executable, and then it will look for its libs in its own directory. Just like is the intelligent default on on you-know-what Redmond operating system everyone rags on.
They've built an entirely statically linked user space for Linux . Until then i never questioned the default Linux "shared libraries for everything" approach and assumed that was the best way to deliver software.
Every little cli tool i wrote at work - i used to create distro packages for them or a tarball with a shell script that set LD_LIBRARY_PATH to find the correct version of the xml libraries etc i used.
It didn't have to be this way. Dealing with distro versioning headaches or the finnicky custom packaging of the libraries into that tar ball just to let the users run by 150 kb binary.
Since then I've mostly used static linking where i can. AppImages otherwise. I'm not developing core distro libraries. I'm just developing a tiny "app" my users need to use. I'm glad with newer languages like Go etc... static linking is the default.