The "problem", in the sense that it's a problem, is that these languages and related technologies are all pretty good! And so it's understandable that many developers who would traditionally be in the open source ecosystem want to use them. As a result they end up creating software that can't easily be shipped in traditional distributions. Ecosystem fragmentation is the unavoidable result.
* By typical open source developers, I mean the sort of developers (and their development practices) that produced most of the software on my computer. I don't mean Firefox: Mozilla and Google have much more standard corporate development practices despite both producing quite a bit of open source software.
Yes; and frankly the development ecosystem for making software for linux & friends on top of apt/etc is terrible - at least from the perspective of a modern professional software engineer. The assumption is C, and C programs have no package manager - so of course dependencies get bundled / vendored sometimes when the alternative is linking to potentially out of date dependencies in apt. Autoconf/automake is awful to learn and understand. CMake is better - but its horrendously complicated because it tries to solve the impossible job of paving over all the junky custom compilation scripts that came before. (And it still has no cargo equivalent for actually fetching your deps.)
And then to work around all of that, each distribution will make weird, custom, maybe buggy patches to your software before adding it to their package managers. (Which has caused some high profile bugs and security issues a number of times.) Now when there's a bug, nobody knows who's fault it is!
This worked in a world when there wasn't much software, when releases were rare and when most programs only had one or two dependencies. None of these properties are true any more.
Rust, go, python and nodejs don't fit well with linux's package managers. The obvious alternative would be putting every crate, gem, pip package and npm package into apt, rpm and all the rest. And keeping them up to date with every version. But lets be real - that would be horrible. Apt et al aren't (currently) up to the task. (Can you imagine every npm package needing a maintainer in apt alone? I can just imagine the github issues: "I'm in debian stable and this transitive dep you're using only has version 0.1 available, from 6 years ago. What do I do?". Yikes.)
I'm sympathetic to the argument that modern million dependency software development has its own problems; but right now it (sadly) has no competition in terms of ergonomics and build reliability.
I don't think this is even the problem.
It's that upstream maintainers have stopped worrying about compatibility.
Once upon a time you would have regular minor releases of some package. 3.0.2, 3.0.3, 3.0.4, but they were all backwards compatible. If you had version 3.0.4 and some software that was built against 3.0.2, it still worked against 3.0.4 because the only difference was that things were added or compatibly improved, not removed or incompatibly changed.
Version 3.0.x wasn't compatible with version 2.9.x, but then the package maintainer for the distribution only has to package versions 3.0.4 and 2.9.16, i.e. suitably recent minor versions of each compatibility revision. Compatibility revisions so old that nobody relevant uses them anymore can be ignored, so they only had to package two or three incompatible versions which together are compatible with everything in active use.
The problem today is that everything is a compatibility-breaking change, so there are dozens of releases from this year alone that are all mutually incompatible and would have to be packaged separately. And that doesn't scale.
I'd also disagree with the idea that the alternative is reliable—try building and modifing a project that hasn't been touched in 6 months, and see how reliable that is. With your tree of dependencies (absent specific projects which abstract over a set of unstable dependencies and hence implicitly stabilise them, e.g. SDL) the stability of your project (whether it is a library or application or framework) is set by the least stable dependency you have. Increasing you dependencies increases the risk, but if you have an ecosystem which values stability, larger dependency trees should not see a significant increase in risk (personally, I'd love to see the scientific rust ecosystem achieve similar stability to python).
That's not to say linux distros are perfect (change can be slow ;)), but there's lots of little things they do get right (how many projects handle updating configuration files correctly—that kind of thing is built into distro tooling), and they enable ecosystem-wide changes more that the alternative (e.g. https://reproducible-builds.org/).
It's relatively easy to convert between packages between different packagers/distros. Automatic converters are exists for deb/rpm/pip/cpan/ctan/cargo, so it's easy to convert all existing packages into one packaging system and drop all of them into huge mono-repo.
Yes, it's much easier to throw a new version to repo in many non-linux repositories. It's the equivalent of rolling distros, such as Arch, or development version of distro, such as Rawhide in Fedora or Sid in Debian. However, it also much easier to: break the world and make into news, distribute keyloggers and steal passwords and keys, forget to backport security fixes to users of older versions, pivot into completely orthogonal thing, etc. No code reviews means no responsibility.
There’s a reason people are turning to docker - because the “portable executable” idea on Linux is so often broken by weird incompatibilities between libc versions or some important dependency being missing or broken from some users’ systems. Automatic package translation suffers the exact same problem - a dynamically linked binary you build on your computer often won’t run on my computer.
Linux has perfect backward compatibility, I still able to compile and run 70 years old app, developed on completely different OS and processor.
IMHO, you think that your app/lib binary will work on all combinations of OS/processor flawlessly without recompilation, which is not true. Nobody promising that. It's by design.
Care to provide some examples?
I know it probably seems like last week, but that was more than twenty years ago.
Just to pick Go as an example (not to be lost discussing VMs and such), it doesn't matter if I am targeting bare metal, Linux, Windows, IBM z/OS, AWS special cloud runtime, whatever.
As long as the Go code is the same, and someone has done the low level runtime support, it is a compile away and done.
Finally by pushing containers no matter what, the Linux community has made this even easier.
That is what I have been doing the last 20 years, in the context of C++, Java and .NET.
Every package installed with Nix is isolated into content-addressable* directories, so for example, my install of Firefox is located at /nix/store/c7pmng2x05dkigpbhnjs8fdzd8kk31np-firefox-85.0.2/bin/firefox. This is pretty inconvenient to use directly, so Nix generates a profile that symlinks all your packages into one place (eg. /run/current/system/sw, ~/.nix-profile), and then environment variables like PATH can just include <PROFILE_DIR>/bin.
With this approach, I can have multiple versions of the same package installed simultaneously, without them conflicting with each other. Like in a traditional distro, any dependencies that are shared between packages aren't duplicated, but if a package needs to explicitly depend on a different version, it can.
Also, because Nix is designed as a functional package manager for building packages from source (even though it has a binary cache), you can trace back exactly what sources were used to build your package and its dependencies, all the way back to the bootstrap binaries used to build any self-hosting compilers (gcc, rust, openjdk, ...)
* Most packages use a hash that's generated from the inputs used to build it, rather than the output that's generated.
At least for the first half, that sounds sort of like Fedora/Red Hat CoreOS[0], the predecessor CoreOS fork Flatcar Container Linux[1], or the Amazon distribution BottleRocket[2].
[0] https://getfedora.org/en/coreos?stream=stable
[1] https://kinvolk.io/flatcar-container-linux/
[2] https://aws.amazon.com/blogs/aws/bottlerocket-open-source-os...
Believe me, it's usually the opposite.
Lack of proper releases, testing, and versioning results in unending checkout-fu to figure out what commits for each of 20 libraries will work for each other.
The idea is plainly stupid, without any redeeming qualities.
The entirety of this cargo cult hinges on the point of "If people calling it a genius invention for last 8-10 years admit it not being it, some major reputation, and cred loss will be incurred"
Because tracking dependencies in source control (specifically, checking in lock files) is tremendous for reproducibility. It means that when bisecting through the commit history to find when a problem began is not just bisecting through the local source code, but also the specific versions of every dependency.
So regardless of whether the issue you're investigating is in the package, a dependency, or an unexpected interaction between the two, you're able to find the first commit that introduced the issue in O(log(commits)) time, rather than needing O(commits * (num dependencies * dependency versions)) time.
Try the oldest and newest compatible version of your code and the oldest and newest compatible version of each dependency. This is O(num dependencies), which is generally small N and you can often guess which to try first. Now do each version of the code or dependency that actually made a difference. O(log(N)) on the versions of that code. You're at O(num dependencies) + O(log(N)), not O(commits * (num dependencies * dependency versions)).
Also, doing it the other way can often lead to frustration. The actual bug is introduced in commit #1234 but is timing dependent, then a dependency is upgraded from version 2 to version 3 in commit #5678 which tickles the bug, e.g. the new dependency version moved around some cache lines. Now you're looking in entirely the wrong place at an innocent dependency.
Whereas if you notice that the bug exists with dependency version 3 and the latest commit and then do binary search on all the commits while holding the dependency at version 3, plausibly the bug shows up between commit #1233 and #1234.
From there you can start stepping through the code, or simplifying the situation to get a minimal repro of what's going on, or even bisecting with dependency version 3 held constant to see if that's diagnostic.
In my experience, with a bug that's so timing sensitive that some cache lines moving around will trigger it, you're likely to discover pretty quickly that something weird is happening as you try to get a more minimal repro.
Meanwhile, if you're not tracking dependency versions, such a test is likely to appear weirdly flaky. You're trying to track down this failure but something else comes up and you have to set it aside for a few days. When you come back to it, you can no longer get the test to fail because an untracked dependency change has moved cache lines around again. Which change? Which dependency? Since it's not in source control, you've got a painful process of guessing plausible combinations of recent versions of all transitive dependencies.
The trouble is that this will tend to point the finger at large changes that jostle many things around at once and become a rabbit hole rather than the two line commit with a typo that actually caused the problem.
The main advantage you're putting forth is to know the versions of each dependency needed to reproduce the problem. But you can get that from the person reporting the bug. You can add a switch to your software to output the versions of every dependency it's using and then it's there in the bug report. And once you have a combination that can reproduce the bug, the process of experimenting with things to identify the cause is basically the same either way.
Admittedly I'm basing this on my experience in a very large monorepo environment, but there's no figuring out which commits will work with each other. Every commit with every library will work, otherwise it doesn't get committed. Yes, this involves massive CI infra and tooling to aid in refactoring.
You want to make a breaking change to a lib? Great, it's on you to update every piece of code that calls it.
It's great when you can control every piece of your infra, but I totally get how it's unfeasible (and maintainer hell) for the distro community.
K8s, go and even ruby, tend to change and evolve. It's usually a bad idea to pull such software from distros then, even if available.
The means to get software is simply too different, and it's a non-problem for everyone but completist distro maintainers.
However it is incredibly expensive on the producer side.
Package managers have pushed that pain to consumers. It isn't terribly surprising that producers are jumping on board en-masse.
This is where the disconnect is coming from. The distro maintainers are coming from a world of multi-user systems where backwards compatibility and updating deps without disturbing a user's workload / forcing them to recompile is paramount.
Go (and a fair amount of rust/python work) come from the land of CI/CD and, to a lesser extent, monorepos. When you are rebuilding the world above a bare minimum of the OS literally on every commit (or at least several times per day), it's easier to reason about code that is running if you can look at the commit that a bin was built from and know exactly what's inside (including all deps).