Shared libraries have always been broken in Linux. Unfortunately many things like GPU drivers, graphics libraries and NSS need shared libraries to dynamically load certain runtimes (because you don't want to load all possible GPU drivers in existence to your RAM). So an ecosystem has been developed on top of terrible ABI and architecture GNU/glibc provided.
C++ abi should not be included in this. It is independent from the other pieces and historically a source of incompatibility on its own.
Saying "C/C++ abi" as if they are the same is looney tunes, the former is very simple and stable and the latter is very complex.
How libstdc++ initializes global variables absolutely depends on glibc and ld-linux.so. That is part of C++ ABI.
Somehow most of my portability issues seem to be caused by glibc, its symbol versioning and close ties to the dynamic loader. Minor versions aren't compatible, no two Linux distros ship the same version and you can't just provide your own without also patching in your own dynamic loader.
At least as far as the defaults on Linux go I consider C the root of all evil.
In which case as long as you're using the documented public API and compile your program with the oldest version of glibc you want to support (some Ubuntu from 4-5 years ago should cover pretty much every current desktop) you should be fine. And with something like Docker this is trivial to do.
Sure it is annoying that you cannot use your current distro (especially if you use some rolling distro) to make binaries for everyone, but it takes very little effort to work around that. The only issue i can think of is if you absolutely want to compile using the latest version of your compiler and you cannot build the compiler from source to work in the Docker (or whatever) contain to work against the older glibc.
In complex cases, it turns out that the old version of glibc also pulls in other libraries and the compiler, and you're stuck with a very ancient sysroot. You may often find that you can't compile new library versions in such a sysroot and link them statically.
So, it looks good on paper, but forget about the ravines.
That's currently the real core of the problem.
The loader (and libdl) need to be decoupled from the glibc itself under Linux.
Without that, any attempt to ship static binaries (or any binary with a different Libc) will be a source of perpetual pain.
nss plugins and its associated pain (sssd and avahi) are an other examples of that.
But I do agree with you, the glibc is all one giant codebase shipped in one giant blob, dispatch under several names.
Even libdl.so is barely more than a wrapper and some glue around the libc.so itself.
C++ global/static variable initialization depends on the specific version of glibc (they don't usually break compat, but they can and they did in the past) which also provides ld-linux.so that loads those global variable placeholders in the correct manner such that glibc and libstdc++ can initialize them correctly.
This is just one example. Thread local variables and behavior of things like pthreads with signal, fork etc all depend on glibc.
And that's the correct approach and also one that many have taken. We just need someone willing to maintain that as an easy mode SDK for everyone.
Who said that? This approach has many problems that have already been discussed here, not to mention the fact that it leaves Alpine and Bionic-based systems out in the cold.
Let me remind you that Bionic is the most widespread libc in the Linux world, and Alpine is the most popular Docker layer.
The world doesn't end with glibc. And it doesn't begin with it.
Until you define a thread local variable (C11) or use atomics (also C11) or define a global with an initial value. Then it happily generates code that depends on "whatever my target glibc + ld-linux.so needs".
The ABI is strongly dependent on explicit libc implementation in current Linux systems. There is no libc independent ABI on Linux.
When you compile libc, you also get a binary loader ld-linux.so with it. They are not two independent components of a system.
Basically all .so files compiled with glibc require the ld-linux.so that's also generated by that glibc (or a later version, if they didn't break the binary compatibility).
There are a lot of stuff that's executed by ld-linux.so and glibc that are not explicitly documented but they are absolutely necessary for your program to start and correctly initialize things like global variables or signal handling or loading other dynamic libraries. Some of that functionality sits in ld-linux.so and some of that in glibc. They have circular dependencies to each other. glibc expects ld-linux.so to put things in certain order but ld-linux.so also must load glibc first to have access to certain APIs. They are not part of System V ABI. They are not documented.
Musl maybe can implement this but it is simply reverse engineering what glibc did and then playing a game of cat and mouse. There is no independent ABI standard.
Then, what is the exact reason a library compiled against glibc must be loaded by a specific ld-linux? I could see that this is true for C++ perhaps, or when you use very special features, but I do not see this for C.
I often compiled programs against one version of glibc and run it against a different version, so I know there is not a tight coupling. So please be specific in explaining in what scenarios this would break.
> Then, what is the exact reason a library compiled against glibc must be loaded by a specific ld-linux? I could see that this is true for C++ perhaps, or when you use very special features, but I do not see this for C.
You still have functionality like `dlopen` with C or `pthreads` with C. ld-linux.so is the thing that prepares the stack frame, or the segment pointers (FS_BASE on x86_64).
When you have your main loaded __libc_start_main_impl is called from glibc and if you disassemble it you'll see this line:
call 0x7ffff7c28790 <_dl_audit_preinit@plt>
Guess where _dl_audit_preinit lives?
(gdb) info symbol _dl_audit_preinit _dl_audit_preinit in section .text of /lib64/ld-linux-x86-64.so.2
Moreover glibc has "magic" sections like .init_first. Only ld-linux.so that is compiled from glibc source knows how to handle that. https://elixir.bootlin.com/glibc/glibc-2.44.9000/source/csu/...
> I often compiled programs against one version of glibc and run it against a different version, so I know there is not a tight coupling. So please be specific in explaining in what scenarios this would break.
It didn't break, since glibc hasn't broken backwards compatibility of ld-linux.so and glibc combination lately. Last time they broke it was in 2024 for a really specific subset: https://sourceware.org/pipermail/libc-alpha/2024-December/16...
A breakage hasn't been observed doesn't mean that there is no tight-coupling between ld-linux.so and glibc that is compiled from glibc source.
If you want a really specific scenario:
- Write a very simple C file that contains a global (volatile if you want to stop the compiler optimizations) thread_local variable with C23 syntax
- Compile a simple .so file on a GNU distro targetting glibc ABI with GCC (pass -std=c23)
- start up a Musl distro (e.g. Alpine Docker image), copy that .so file in
- Write a C program that dlopens the GNU .so file
- Try accessing the thread_local variable you defined
- Watch the world burn
It's not Musl's fault that Glibc has an undocumented, badly designed, tightly coupled implementation. It's not Musl's fault when they decide to not follow Glibc's design which they have to reverse engineer and rewrite from scratch. Then Glibc can break it in a future version anyway. They give 0 guarantees and have a track record of breaking things.
Sorry but I feel like you're trying to excuse Glibc out of their terrible design. Glibc didn't ask anybody for standardization. Glibc didn't document their behavior. Musl doesn't need to follow any undocumented behavior. Musl has its own implementation of thread_local variable and it works only for Musl dynamic binaries (which you cannot load with Glibc's ld-linux.so either). You need to tell your compiler to generate code that Musl prefers not Glibc. So effectively we have two ABIs.
There is no standard and any design that tightly couples the system binary loader to the C standard library is just extremely terrible design. Both Musl and Glibc do it. Both of them are terrible. Glibc was first and it set the horrible, binary hostile behavior "have no standards, trust Glibc, and recompile your programs again" as the "standard".
The entire Linux userspace depends on Glibc's terrible design. Due to Hyrum's Law, the exact needs of the complex and emergent behavior can only be surfaced via making a better designed, completely independent system loader and a completely independent libc and then fixing the entire tens of thousands of userspace libraries in the upcoming decades. It is just Sisyphean work to fix Linux userspace.
There is no simple "let libc-X obey the standard S" solution. There is no standard. Linux Standard Base project tried to set a standard. It failed.
I'll won't further discuss this with you. Maybe you have good intentions, maybe you're playing dumb. I'm not sure anymore. If you're the former, I'm not going to deliver every single bit of information. I provided enough resources and you can learn.
But, thanks for spending the time! I learned a bunch!
I'm not arguing that there aren't problems here, just that I think people have different expectations that are a little extreme. The libc implementation is a core part of the operating system, if you want to target that OS you should target that libc. Some distros (alpine) prefer musl libc, most prefer glibc. And distros really are different operating systems.
Not really, though. glibc uses symbol versions that are forward but not backward compatible. If you got an error that said "this program was built for a newer version of <distro>" would you say the same thing?
Note this is the same (if not worse) on MacOS, and on windows you used to distribute the CRT with your application just to deal with the same problem.
Yes glibc has some backwards compat but you cannot load a binary compiled with a newer version of glibc using an older ld-linux.so. That's because the interdependency. Nor you can load binaries that depend on different libc.so files with glibc systems
I cannot comment on macOS, I have never used it. However this is not a problem with Windows. You can ship a newer CRT or you can install it as a system component using Microsoft's MSI. The dependency is one way on Windows. CRT purely depends on Win32. Moreover the loader is completely independent and DLLs are loaded into their own unique scoped namespace unlike Linux that loads them in global symbol namespace. That's why you can mix and match DLLs compiled for different CRT versions.
On Windows you don't need to ship a new binary loader. I can just ship Windows 10 UCRT DLL (which is the new libc of Windows) to Vista and my binaries will work. The binary loader isn't interlinked with the libc.
One of the ways Windows manages to support multiple libc's is by being careful not to mix allocators; if a system API you call allocates on your behalf, your libc can't free it, the system API will offer a function to free it.
Windows loader is certainly available to user programs though; LoadLibrary has been around longer than many developers.
As for older Windows systems not being able to load new DLLs, they can; the format hasn't changed in a very long time. I've had experience with installing some DLLs on Windows NT 3.51 and running a modern Firefox, which is about 20 years behind the times.
MS haven't made this work arbitrarily far back, I believe they deprecated targeting Windows XP in one of the more recent toolchains, but that was purely a "not worth supporting" situation.
EDIT: mixed up talking about older & newer
csrss.exe handles some of the loading functionality: https://en.wikipedia.org/wiki/Client/Server_Runtime_Subsyste...
What happened to the Unix principle "do one thing and do it well"? Windows solved this quite modularly and it works perfectly well.
Glibc is terribly designed and it is the core component of almost any Linux userspace app today. Let's not try to find excuses for horrible designs.
The reason they're tightly coupled is because there needs to be a bridge between Sys-V ELF ABI and the C runtime, and it makes sense for the implementation of that bridge to use libc itself which means special-casing the bootstrapping of libc during program loading.
glibc isn't unique here btw - musl libc ships its own loader that does the exact same thing as rtld (parse auxv/envp off the stack, relocate itself and load libc, initialize the CRT, then load the rest of the program's SOs in order).
Even if they were fully decoupled it is not unreasonable for software targeting newer versions of an operating system to only run on that version or later. Linux is actually the outlier here where the syscall interface is stable enough to make this a non-obvious problem.
There's a universe where you can load multiple versions of the same libc into the same address space but that isn't ELF, and it's not common. aiui the only reason PE can support e.g. multiple versions of malloc/free in the same executable is due to two level namespaces in symbol relocation tables.
---
edit: another observation is that for a functioning ELF loader you basically need a three stage bootstrap. Stage one parses the stack passed by the kernel and performs self-relocation. You need that to make non-static function calls and read/write global variables. Stage two loads libc and initializes the CRT with stuff like environment variables, aux vector, any dso for syscalls passed by the kernel, etc. That has to coupled because POSIX specifies stupid stuff like "getenv" and "argv" requires the loader to know how to move from the stack passed by the kernel to the memory reserved by the CRT that gets used by the final executable. Finally in stage 3 it can relocate and load shared objects and run any constructor functions pre-main, before jumping to start where the precompiled CRT start (then jump to main goes).
So if you want an ELF loader on Sys-V you probably need to couple the loader to the libc, just for libc to work at all. Unless you want to specify a bunch of behavior and binary layout in the C standard.
None of this is designed by glibc, it's the simplest way to go from exec to main.
I've become obsessed with getting rid of it, especially after I realized that contributing to GNU itself was a dead end. Freestanding Linux programming turned out to be much more fun anyway.
All libraries out there should adopt the SQLite design: programmers provide it with all the necessary functions. Instead of libraries hard depending on glibc, we get to inject the libc-ish subset it needs. Then we can use whatever we want under the hood. I'm working on porting SQLite to freestanding Linux system calls so it can run with zero dependencies. Wish I could say the same for software like mesa, I'd need a lot of help for this one...
Can't be done. The libc is legacy, it can't be changed without breaking everything. It's also mandatory on every operating system other than Linux.
> what is necessary for users of the C language to actually do stuff
Surprisingly little. I wrote an entire lisp interpreter in freestanding C with Linux system calls. It managed to survive for a rather long time without any memory allocation at all.
The system layer is refreshingly tiny. It consists of a memory allocator and extremely basic functions like memmove and strlen. I successfully got rid of total nonsense like thread local errno, locales, implicit buffering, cached global state, possibly more. All that stuff is gone!
Great choice for small programs, but what if I want hardware accelerated 3d?
With Linux system calls alone it should be possible to set up kernel mode setting without depending on any toolkit at all. This should be enough to get a framebuffer for software rendering.
For hardware acceleration though, one must give this graphics context to an OpenGL ES implementation. That's where it gets ugly. There is no way to divorce that from the libc short of literally rewriting it.
Maybe Vulkan will enable it? I can't say for sure at my current knowledge level.
https://vulkan.lunarg.com/doc/view/latest/mac/LoaderInterfac...
Windows explicitly does not want you to link the system libc. You are expected to bring your own, and doing so means your process has multiple libc's loaded into its address space.
And if you choose to build a binary that doesn't need a libc, you won't be bringing one.
This is categorically false; UCRT[1] is a thing. The 'U' stands for universal. Unlike Linux, Windows allows developers to choose their ABI boundary and also ship that boundary if they desire, or use the 'system' one and ask older platforms to install redistributables or Windows update packages. There's the old and new C runtimes in MSVCRT.DLL and UCRTBASE.DLL, the C++ runtime in VCRUNTIME140.DLL, Win32 in KERNEL32.DLL, USER32.DLL and more, and then the stable-ish kernel interfaces in NTDLL.DLL, in order of 'closeness to the kernel'.
And also, 'libc' is a UNIXism; on Windows the term is CRT, for 'C runtime'.
[1]: https://learn.microsoft.com/en-gb/cpp/porting/upgrade-your-c...
Usually Windows developers will use Win32 directly, ZeroMemory() instead of memset(), and so on.
> At some point, the decision was made to just give up and declare it an operating system DLL, to be used only by operating system components.
https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=12...
When you link with UCRT your specific program just gets a "view" of standard C functions. You can load DLLs that were linked with MSVCRT from a UCRT program. It just works
amd64: https://devblogs.microsoft.com/oldnewthing/20260319-00/?p=11...
arm64: https://devblogs.microsoft.com/oldnewthing/20260320-00/?p=11...
Linux has a stable system call ABI. I can trash the entire user space and boot Linux with init=/my/program if I want.
It's not that Linux doesn't allow developers to choose, it's that Linux doesn't actually control the userspace ABIs. Userspace is whatever we want it to be, and people settled on "GNU/Linux".
This is a noble but practically worthless ideal. Hardly anything is written against the kernel ABI except glibc itself. End-user application developers don't write against the kernel interface. Platform library and application developers like KDE and GNOME don't really write against the kernel interface either. Nowadays even driver developers don't entirely write against the kernel interface, instead also linking in glibc and lib{std}c++ for things like shader compilers (which is nowadays usually just a fork of Clang). Like it or not, glibc is the singular entry point for desktop Linux, which is the point of essentially every comment under this post.
That 'desktop Linux ≡ glibc' is something that I daresay the kernel developers also have to take responsibility for, rather than shrugging their shoulders and pawning it off to GNU. Linux's unstable user-mode is the entire reason for Docker's existence. It is bodge upon bodge and has led to much gnashing of teeth. The GNU/Linux community (I don't care for Stallman's meme rant here) have to grow a backbone and understand that an OS is much more than just a kernel, it is a platform, and currently the Linux platform is a disparate set of communities with highly divergent goals.
That's why I've always said the best Linux is Android, because Android is a platform.
I'm working hard to change that.
Here's an entire lisp interpreter written in freestanding C using nothing but Linux system calls:
https://github.com/lone-lang/lone
It has a system-call primitive too, which means it can do anything.
Here's a TCP echo server written in lone lisp:
https://github.com/lone-lang/lone/blob/master/examples/tcp-e...
It works, and you don't need libc.
> Like it or not, glibc is the singular entry point for desktop Linux
Well I don't like it, and I've made it my mission to fix it.
Maybe lisp isn't the answer, Rust is. I've revived my liblinux crate too.
https://github.com/matheusmoreira/liblinux
I'm going to add system calls to this thing every day until I have every single one. Then I'll use them to make all of my applications.
Already have my first project planned: a custom Rust network stack for my QEMU virtual machines so I can firewall them off in userspace without root access.
> the Linux platform
The linux platform is these system calls. Everything else is userspace. This is exactly what makes it special.
This is exactly the attitude I meant by 'the kernel developers also have to take responsibility for, rather than shrugging their shoulders and pawning it off to GNU' and 'an OS is much more than just a kernel, it is a platform'.
By 'platform', I mean windowing toolkits, such as Wayland. Audio libraries like ALSA and PipeWire. Desktop idioms and libraries like DBus, and the entire freedesktop.org effort including Mesa and more. Init systems like systemd. Bootloaders like GRUB/systemd-boot. Networking, including systemd-networkd, iwd, and NetworkManager. Power and device management including PowerDevil, Bluez.
If all of this is considered 'user space' then my point is strongly vindicated. On most competitor OSs these comprise the 'platform'. You won't ever encounter a Windows or OS X developer arguing that USER32.DLL or Foundation is 'userspace' even if it is technically true and code from those libraries may only ever run in ring 3. From the average user's point of view, the only 'userspace' are end-user applications that they have installed, like Chrome, video games, etc. From that perspective, even purely user-mode applications that ship with the distro ought to considered 'platform'. This includes such trivia like calculator and camera applications.
We need to talk about the practical realities of a platform, not a theoretical, unattainable, and as I mentioned, useless ideal. These practical realities mean shipping a unified set of useful 'batteries included' user-mode libraries and applications. If one wants to develop the platform itself, then sure, the kernel has a nice stable interface, but on most Linux distributions the kernel is built from source anyway, so even that nicety is moot.
You mean Linux distributions.
> From that perspective, even purely user-mode applications that ship with the distro ought to considered 'platform'.
Every Linux distribution is its own platform.
You decried Stallman's meme rant yet you just rewrote the entire rant in your own terms. Linux is the kernel, GNU's just the stuff on top. You can get rid of it if you want and make your own. Android did exactly that.
Also, I daily drive Alpine Linux (using musl) and I'm quite happy to assure you that glibc really is only one option, not at all mandatory for desktop Linux.
I never disparaged it; I said having it doesn't make it any more useful that OSs that don't, because the practical reality is that the kernel interface is narrow and highly technical, and a stable kernel binary interface has little to do with a user's day-to-day experience with an OS.
I daresay Android would still work pretty well even sans the kernel ABI stability, because the kernel is usually built from source, and if it isn't, then every Android source tree has kernel prebuilts for the kernel version shipping. I also never said nor implied glibc should be uniquely privileged; I think you have misrepresented my comment.
Some advertisement for VC-LTL: https://github.com/Chuyu-Team/VC-LTL5 and https://github.com/Chuyu-Team/YY-Thunks
VC-LTL Implements a good subset of modern CRT functions using the msvcrt.dll, which makes modern MSVC compiler to emit binaries that theoretically can run in Windows NT 4.1 or Windows 98 (until CFG broke them, but you can turn it off). Still can teach an old dog new tricks.
YY-Thunk is a little more funky, it tries to implement new Win32 userspace API for old system by emulating the behavior and expected values using userspace API in the old apartment, so you can try to implement a function in Windows 10 with functions in Windows Vista, and hopefully it will work out.
That only massively compounds the problem.
> And if you choose to build a binary that doesn't need a libc, you won't be bringing one.
NT system calls are not stable. You still need to link against ntdll.dll at the very least, like a forced Linux vDSO.
The Windows ecosystem, that manages to deliver built binaries easily & widely, regardless of whether the author has a 1 year old OS or a 15 year old OS, suggests that it's not as big a problem as you believe.
I'm sure those OSes make efforts to make said runtime binary compatible between executables.
That's when you run into the Darth Vader of binary interfaces.
> I have altered the ABI. Pray I do not alter it further. -- De Raadt
GPU drivers are a perfect example - GPUs are hellishly complex, and their interfaces are super high-level - OpenGL/Vulkan is not something that should be implemented by the driver.
I think a general practice that is followed with GPU drivers is to move as much of the complex and non-privileged stuff out of the kernel, and only keep modesetting/power management/memory and control stuff in.
That reduces massively the amount of code that needs to run in Ring 0, but in exchange, the interface between the user and kernel blobs is non-stable. Programs talk to the userspace blob and there's no guarantee of compatibility between different kernel/userspace driver versions.
This is something even Linus accepts, with the only contention that the userspace blob needs to be open source as well. They're not maintaining something in tree, that they have no way of testing without proprietary external deps.
Or.. any other programming language. A lot of systems programming is done in Rust nowadays.
Windows never had this problem. If you really want it, you can actually skip CRT completely on Windows. Only thing you need is the initialization code that's statically linked anyway: https://nullprogram.com/blog/2023/02/15/ . You can still make Win32 calls with it. CRT is there as a convenience.
And since Linux distros are compiling everything from source anyway, this makes Linux a prime candidate for complete libc and complete ABI replacement. Thanks to stable kernel ABI, we can also use the same kernel even with Docker etc! Musl did it. Why can a completely modularized libc not?
Will it break a lot of programs in the userspace? YES! Hyrum's Law exists.
Will it be rejected by a significant part of user base? YES! They rejected systemd as well.
Will it take one or more decades to have something working? HELL YES! This is Linux we're talking about. Everybody has seen how Wayland played out. However, maybe if Valve has enough motivation to fix Linux binary problem, we maybe can get finally a binary-compatible distro in a decade or half a decade.
Solo is basically pg83's answer to the architecture you just described. If there is no libc independent ABI, build your own loader and shim the boundary.
I can understand Linus's obsession with taste and the areas it was overlooked or traded.
Or do the MS thing, and ship multiple versions like msvcrt
Guess what happened with LSB? Some distros like Debian never adopted it. It also broke backwards compatibility at random times. It is now abandoned.
> Or do the MS thing, and ship multiple versions like msvcrt
MS can do that since the CRT is decoupled from the system DLL loader and Windows ABI is designed to isolated loaded DLLs into their own symbol namespaces. So your program can load DLLs that depend on a different CRT version.
Glibc's design tightly couples the system loader with the specific libc. Moreover symbols in ELF files are loaded into a single global namespace. You cannot ship shared libraries that are compiled with a different version of Glibc or Musl libc.
All solutions to this problem are hacky, complex and controversal and highly fragmented, where this should be BASIC functionality
It isn't that this is 2026 and it was a bug that hasn't been fixed since. The userspace of GNU is designed against what you consider as basic functionality.
What you can't do is build something statically with musl and then reliably dlopen shared libraries built with glibc.
If you break or remove a shared lib here, you may no longer be able to compile something from source. I had that happen in the past before I started to use more statically compiled programs (and busybox too).
Assuming everything works as-is via shared libraries at all times, makes little sense for ALL linux systems. For instance, some people upgrade glibc manually. Then you need a working base system to resume compilation. I do that for my customized gobolinux system, so I can use any program version as well as any glibc version (assuming I can still compile the program; many older programs no longer compile).
Additionally many other OSes have had both approaches since their early days, Xerox PARC ones.
For some strange reason they assume to know better than all those researchers.
> GPU: Vulkan and OpenGL drivers are supplied by the host as shared objects, usually built against glibc, and a fully static musl binary cannot normally dlopen() them.
Why? Have people managed to break the ancient concept of shared libraries, and this is a fix for that?