I have little Linux experience, so pardon me for asking silly questions. Doesn't Linux then choose a weird place to put the border between "OS" and "userspace"? If, say, an opengl library or an ssl library installed by the distribution breaks backward compat with earlier versions, how is that not the OS breaking backward compat? Sure, if a user hand-installs some new library version, globally in the system, --force, y y y, then I get it. Things will break. But that's not what's going on, right?

It feels a bit weird to see the kernel folks put so much effort into "never breaking userland" only to have the distributions throw that all away with a "yolo, the new libyadayada is 1% faster and who cares about old programs!" kind of attitude. Is that really how this works? Or am I just misunderstanding some nuance? I'm not trying to be critical on anyone btw, I'm just trying to understand what's going on :-)

> Doesn't Linux then choose a weird place to put the border between "OS" and "userspace"?

Your fallacy is assuming that "Linux" is a single entity with a single mind. Linus is very strict about backwards compatibility, but he only controls the kernel, not glibc or Mesa or KDE.

Sure, but that just means a failure of Linux as a product experience.

Users don't make that distinction, even a little bit.

This is also arguably a failure of the open source model in practice (though not in theory). It tends to generate projects that are really good at infrastructure and really bad at product. The ones who fill in the product experience as best they can are the companies that are trying to get paying customers, because those customers won't stand for a weird experience, regardless of the technical/governmental reasons

The open source model expects apps to be open source. And if they are then an app can be recompiled to support new library versions.

It is the hybrid open source library / closed source app that causes problems like this.

Having the source code will not help you when the APIs change - see SDL1 to SDL2, Gtk1 to Gtk2 to Gkt3 -soon- to Gkt4, Qt1 to Qt2 to Qt3 to Qt4 to Qt5, etc and all the desktop environments and tons of applications wasting their time "upgrading" from lib version N to incompatible lib version N+1 because the lib's developers do not care about their users (both in the end user and library user sense) and broke backwards compatibility), just so that these applications will do the exact same thing, just now with a different API.

No, backwards compatibility isn't something you solve with open source, open source is perfectly capable of breaking it in all the fabulous ways possible as has been already demonstrated many times.

Backwards compatibility is something you solve by actually caring about it when you are designing your version N+1 library (and preferably, even when designing your version 1 library's API to allow for it).

SDL1 can be installed alongside of SDL2. Same with GTK and Qt. I really fail to see the issue here. If a developer of a Qt4 application doesn't feel like updating to Qt5, users can still use the Qt4 version with Qt5 installed in their system. Most Linux distributions still package Qt5, Qt4, Gtk3, Gtk2, Gtk1.2, SDL1.2, SDL2...

On top of that, if a binary-only application really wanted to stand the test of time, it could statically link the libraries.

At least Debian has dropped GTK1 and qt3 - it's not trivial to get old libraries on a modern system.

This doesn't help much when the specific application you want to run is not available in your system because it doesn't ship Qt3 or Qt4 or Gtk1 or whatever (note that AFAIK none of these are shipped anymore in Debian sid) and it also doesn't help much if the application you are trying to run is using a library that, even though it exists in version N, there is already a version N+Y (Y=1, 2, whatever) that essentially provides the same functionality to the application but it is backwards incompatible while providing more functionality to the user that you'd like to use.

To make that last part more clear, imagine a theoretical "libopenfile" library that shows a file dialog, this library in version 1 exposes a function like "open_file(const char* path);" but in version 2 they improved the GUI a bit (e.g. making the dialog make better use of screen real estate, added a create folder button, etc) and changed the function name to "lof_select_file(const char* path);" because arbitrary tidyness reasons and in version 3 they changed the signature to "lof_select_file(const char* path, unsigned flags);" and added the ability to load plugins that could provide more actions for each file in the displayed file list, show overlay icons, etc.

As things are right now, a modern Linux system would need to have all three versions of the library installed - assuming the distribution cared about backwards compatibility to provide all of them instead of the latest and perhaps the previous version - meaning there would be three different libraries providing essentially the same functionality in three slightly different ways. Moreover, and also very important, version 1 applications do not get to use any improvements introduced in version 2 and neither version 1 nor version 2 can use the plugins introduced in version 3 - even though, again, all they want is to show a dialog to choose a file.

The way that would have been done right is for the version 2 of the library to still export the "open_file" function but simply make it call "lof_select_file" and version 3 make a "lof_select_file2" (or "_ex" or whatever, if the reason to not do that is because you dislike the naming, you are part of the problem) that accepts flags and have "lof_select_file" call it with a default set of flags. Then everything would work, from version 1 to version 3 and all applications would get the functionality introduced in subsequent versions (and as a sidenote, no, i wouldn't remove open_file from the header files to encourage using the renamed function since backwards compatibility in source form is very important too - not everyone who gets to compile a piece of code is the developer of that code, or even a developer).

Now the above is a bit of a strawman, but it is exactly the situation you get with Gtk1, Gtk2, Gtk3, Gtk4, Qt1, Qt2, Qt3, Qt4, Qt5, SDL1, SDl2, etc and pretty much most of libraries that you find on a desktop system (notable and thankful - so far - exceptions, being the X11 libraries and... Motif, although that isn't installed by default anywhere so it isn't really something you can rely on).

In a worst case scenario, couldn't you simply bundle a static version of an old library with an old open source app to avoid the problem of a distro dropping old dependencies?

This works when it's only libraries that are changing. As soon as protocols change, you're fucked. Case in point, Wayland. Right now, most Wayland compositors run Xwayland (i.e. an entire X server) under the hood to enable legacy applications without Wayland support to run on them. This will eventually not be supported anymore. Not today, not tomorrow, but maybe in 10 years. Then it's gonna be absolute game over for everything that's compiled against X libraries.

In a Linux distro you specify dependencies by library version and you are never required (as far as I know) to upgrade to a new major library revision. Games compiled for SDL1 are not required to upgrade to SDL2.

There are some absolutely ancient games (Xgalaga for example) that I used to play on Linux on 1996 that still play just fine today on my Debian machine.

Games compiled for SDL1 are not required to upgrade to SDL2, but games compiled for SDL1 will also not get any functionality introduced in SDL2 that could have been transparently made possible to them if the API wasn't broken in SDL2 - e.g. supporting alt-tab, wayland, etc.

Xgalaga works because the X11 library developers have not broken backwards compatibility. This is a good thing. This is what i'm referring to when i'm talking about a library not breaking backwards compatibility and what something like Gtk and/or Qt (and, IMO, SDL) should strive for.

EDIT: i give a more detailed example here - https://news.ycombinator.com/item?id=19813074

> Having the source code will not help you when the APIs change

Yes it will! If it's free software, you can update to use the new APIs!

Developer time is neither free nor infinite.

Your personal dev time is as limitless as your desire allows it to be.

Errr

1. users needing to think about recompiling things is definitely not the experience they want

2. It assumes the compiler, runtime, etc don't change API ever. It also assumes the libraries don't change API ever. Or don't change implementations of existing APIs in incompatible ways.

Instead, both happen all the time.

Having source doesn't fix this.

Distros don't fix this either unless the only stuff you use is in the distro (and even then they still get it wrong).

> users needing to think about recompiling things is definitely not the experience they want

I think the intended OSS experience would be for users to get everything from their distro’s repository, whose packages get recompiled by the distro maintainer.

I agree this doesn’t work in practice, but there’s a reason Debian tells people to only ever use packages from their official repos.

I mean I get it, but it's somewhat ironic given how often i see folks making fun of Apple/et al for saying people are holding it wrong, yet if you look at the other comments in this discussion, it's literally the same thing.

> It is the hybrid open source library / closed source app that causes problems like this.

There can exist good reasons to deliver the application as binary even though it is open-source.

The difference rather is: Microsoft cares about binary compatibility while the developers of userland libraries on GNU/Linux do not.

Agreed. As a former MS-focused engineer in a previous life, I was constantly surprised by how many executables just continued to work 20 years after authorship. It's a different set of tradeoffs than other OSs have made, but a really remarkable one.

Gog.com makes their living specifically by updating games that no longer work on modern versions of Windows.

> Gog.com makes their living specifically by updating games that no longer work on modern versions of Windows.

Many old games that GOG distributes are not written for Windows, but for DOS.

Through emulation (DOSBox), in many cases.

Most distros are distributed as binaries.

But since the source is available and the licenses allow distribution, it's much easier to just recompile the thing.

Binary compatibility is not valued if you can just recompile the source. Why waste brain cells maintaining compatibilty when the compiler can bake you a new compatible binary?

Because your users might be actually using more software than what you deem worthy to put in your distribution.

Most, if not all, distros have no vendor-level restrictions on repository choice, you can make your own.

I think you are conflating kernel with distribution.

there's a reason people don't use linux outside of the server space and most of it is every piece of weirdness introduced above linus. the more esoteric your linux flavor(and closer to bare metal it is), the better your linux experience long term.

A dependency is a dependency. The distro does not matter. It matters what the game depends on and what the distro ships.

Or, you know, snaps and appimage and flatpack. Where your only hard dependency becomes Linux. Which doesn't break userspace.

Some early linux games made the mistake of using low level graphics libraries (not one of the opengl levels), these might not work because the newest drivers just don't support it.

Except that Snaps and Flatpack are over-engineered inflexible garbage compared to how this is handled in sane environments. Flatpack can't even handle the concept of installing to different media without building a brand new repository on it FFS (I don't know if this is true of Snap because it is far too Canonical-centric for my tastes). There's no reason it has to be like this.

Even AppImage is slightly over-engineered in my opinion, but otherwise the only real failing of it is that since there's no standard for ELF embedded icons (wtf is wrong with UNIX devs!?) it's a shit-show to get them to show up as anything other than a generic one.

This isn't hard, Linux people. Just standardize on a set of core libraries and stop breaking ABI outside of major versions, then you'll magically have this thing we call a "platform" upon which applications can run without being recompiled.

Your helping hand is welcome! Even coordination of developers toward solution is helping a lot, because we have no free but dedicated managers. We are developers after all.

Create ticket in bug trackers of all major distributions and describe the problem and your proposed solution. If your vision is right and attractive, developers will join the movement. This isn't hard, it's just time-consuming.

I would, except I've seen others do that and they are unsurprisingly ignored or ridiculed. Someone already made a standard proposal for ELF icons, nobody uses it.

Linux open source - you can contribute but on your own fork over there in the dark far away from us enlightened folk that actually make worthwhile contributions to this project. I only contribute on things that clearly have no "stewardship" or vested interests, lots of useful proposals shot down

Wow, very constructive, great post, love it, keep it up.

To the end user none of that matters, all they know is that this thing does not "just work". windows and macos has that, no flavor of linux does(currently). i look forward to the snap/appimage/flatpack war, it's long overdue

Eh, macOS breaks old apps like it makes Apple money.

Like Aperture from what.. 2014? Won’t run on the next macOS.. that’s an Apple application that can’t last more than a few years on an Apple system.

Even Linux desktops are better than that, but not much better. They seem so emotional.

And that was for me the main reason i stopped using macOS, as every new version of the operating system made software i had paid money for stop working - to the point that nothing works now.

Basically, it's a matter of being able to force the issue. Distros are largely volunteer effort and are already a lot of work. Forcing library devs to cooperate would be a gargantuan task and just rejecting the changes would kill most distros (basically no libraries do this). Instead the distro devs just try to fix the compatibility problems themselves but they occasionally miss stuff.

The only reason there are so many distros is because of how ridiculously complicated it is to build a functional Linux Desktop from the component software. If Linux were a more stable platform, you wouldn't need distros (or package managers for that matter).

Welp this is where the issue and the meme is. This is where the Stallman rant: https://www.gnu.org/gnu/linux-and-gnu.html came from and is still a point of contention. The linux kernel is the only standard part of any distro. Anything beyond that is optional however it will most likely be GNU coreutils. Even then it may not be standard as coreutils may be replaced by busybox or BSD utils or any other core Unix utilities package.

So that's what we are left with. Linux is not an operating system. GNU/Linux is, Busybox/Linux is, and all the distros are operating systems but Linux is not. people just like to clump all these different operating systems together and pretend that they are different versions of the same OS.

If you want a "Linux" operating system that is well defined, look for the ones without Linux in the name. The BSDs have actual UNIX support (i.e. a standard library of tools) and Guix is a Unix-like/Linux-like that focuses on reproducibility and dependence stability.

It's made worse by the common UNIX developer insisting on hard-coding fixed paths into things. The whole ecosystem is just plain hostile to the idea of applications that aren't recompiled every time something changes.

Got a list of such software?

Serious question; doesn’t static compilation do the same thing essentially?

Kind of, but I guess the other solutions are slightly better at accomodating services that are more than a single binary. I think static binaries really should be evaluated again as they seem to be written off by most big players. Though Go(lang) has preferred static linking which has been something that seem to have been appreciated.

Static has its own issues. GPL vs LGPL forces a lot of apps to dynamic. Also security vulnerabilities in libraries become unpatchable.

I’d rather be able to run something that’s security flawed than not run it at all. Those security issues in libraries are probably not such a big deal anyway, because you can run Unreal Tournament 1 multiplayer now and not immediately get hacked, I’m sure. Maybe I just got very lucky.

Apps are always the most secure when they can't run at all because of ABI changes in their dependencies.

Of course as far as graphical desktop software goes, 99% of the dependencies are in the userspace. Unless the userspace libraries (or parts of it) decide to follow Linus' mantra of not breaking the applications that rely on their code, kernel not breaking userspace isn't very useful to graphical desktop applications.