For Rust, these can also impact build times so people actively work to keep down what dependencies can be duplicated which are semver incompatible versions.
I had a privilege to use conan on semi-large codebase - what a clusterfuck that is. I can't imagine a single scenarios why I would be using it.
For example we're talking in English right here, and it's a rich language we can express a whole lot in. It has no version although it's slowly updated over time. But it's mostly the same version around the world, with some local differences. We still understand each other. Despite our minds may be vastly different, we can express each other in this language. Same goes even for language models now.
Imagine we had something as common as JSON, but more in-depth so it can be used as direct representation in memory, and this was the lingua franca not only between servers, but between libraries, apps, and even objects within an app.
Then our dependency entanglement issue would also not exist.
What we did was a choice. It was a lazy choice, ignorant choice. A choice that introduces instant tech debt. And dependency conflicts are one of the results of this choice.
Android has API levels where they don't do that, but that's because Google can afford to provide almost everything themselves.... random npm and pypi contributors don't seem interested in keeping compatibility.
So some kind of smart version system or container or something seems to be needed.
Although, I think in a perfect world our standard libraries would be rich enough that we would not need many libraries, and the ones we did need would mostly only be a few hundred lines at most, and we'd basically never have libraries depending on other libraries or any tangled webs like that.
That said, I don't understand this mentality that libraries can never change. Eventually things need to change. It's impossible to predict all future requirements.
There are and have been several attempts to do things like this. One of them is the Arrow library for "arrays" as used in scientific computing, machine learning, etc. It's a huge amount of work.
This is utter nonsense. If you think you know how to do this better, there are billions or trillions of dollars waiting to reward you.
Intelligent creatures like humans can fairly easily compensate for communication mismatches. That's enormously more difficult for us to program computers to do.
We've only just stumbled on technology that might help with that - the latest generation of language models - but it'll probably take decades of work by everyone involved, on the entire planet, to be able to get to a point that resembles what you're imagining as the non-lazy, non-ignorant choice. Which proves that laziness and ignorance are not the issue.
But this gets more challenging as the granularity of what’s being packaged reduces. You can see this even in the container case: consider a container that depends on a particular version range of another container. For any possible solution to this, there are disadvantages which make it untenable in some situations.
There’s no good general solution to this other than being explicit about the version dependencies and solving the dependency issues when deploying, whether manually or automatically. A automated solver helps with that.
Btw, you can see something like what you described in Java programs built with Maven (for example). Run Maven’s dependency:tree function on any non-trivial program and you’ll see different versions of the same package appearing over and over, because each package does have its own (logical) space for dependencies.
But Maven still has a version solver built in, and even with your proposed solution plus a solver, you can find plenty of articles on the web saying things like “Maven dependency conflicts can be really hard to solve.” The reason for that is when it fails, it’s because it’s encountered a difficult or impossible situation that (currently) requires human intervention.
The way Android and more recent Windows do things is my favorite solution so far.
All dependencies included in static packages, not actually containerized, and with all the heavy lifting done by things built right into the OS, always there, and specifically designed to be used that way, without much emphasis on lots of separate daemons, at least not on desktop.
Even just saying "The OS always has this daemon" isn't enough, because so many services are designed to be configured specifically for a deployment and there isn't any one size fits all solution.
Even something as simple as MQTT has a ton of ways to configure it, and there's no way for a client to even query what permissions it has and what other clients are on what available servers, it all has to be manually set up.
Could it be done better? Sure. But there are tradeoffs. You mentioned Android, where the obvious tradeoff is that the entire ecosystem revolves around the Dalvik VM. That approach isn't really practical right now for more general OSes like Linux and Windows.
Obviously it's a problem because of all the millions of man hours of existing stuff for other OSes, but in theory, I don't see why a managed code only ecosystem wouldn't work.
I could even see it being faster than current OSes, in some future imaginary world where Android had a MakeHardwareAcceleratedCDNServer(folder, port, config).
So many applications(Obviously not all, but a fair number) out there are fairly boring, probably shouldn't even be written at all, and probably could be covered with a small set of high level primitives.
If we had kept on thinking the way we did in the early days of Windows, we'd probably have more things like Excel, abstractions that cover insane numbers of use cases without plugins or anything.