Ruby works the same way, but bundler dependency manager solves it anyway to give you per-project dependencies not just system-wide dependencies. (I believe other well-liked dependency managers like cargo are largely based on bundler's semantics).

Perhaps ruby was more "hackable" by bundler. (Bundler has now become part of ruby stdlib, but didn't start out that way, it definitely started hacking around the way the more fundamental stdlib 'rubygems' worked).

> Ruby works the same way

Kind of, if you ignore Rubygems, which is also part of stdlib at a lower level then bundler (and also, originally wasn't.)

> but bundler dependency manager solves it anyway to give you per-project dependencies not just system-wide dependencies.

It can do that because rubygems manages multiple installed versions of packages and allows per-project ("per call to require", potentially, IIRC) specification of which one to pull from the globally-installed versions (this was originally done by monkey patching require when rubygems was an add-on.) This lets bundler easily live on top of it providing per-project dependencies somewhat more smoothly than Rubygems does without requiring anything like a venv.

> Perhaps ruby was more "hackable" by bundler.

Ruby is ludicrously hackable, yes.

> (Bundler has now become part of ruby stdlib, but didn't start out that way, it definitely started hacking around the way the more fundamental stdlib 'rubygems' worked).

Rubygems also wasn't part of stdlib originally, and started out relying on hacking around the way Kernel#require works.

> It can do that because rubygems manages multiple installed versions of packages

Oh wow, the default python dependency manager only lets you have one version of each package installed system-wide?

Yeah, that is a limitation. As opposed to rubygems (the first dependency manager although as you say not originally built-in to ruby) which has system-wide install, but always let you have more than one version installed.

Without fixing that one way or another, there's no sensible way, true. virtualenv is certainly one way to fix it. I wonder if there would have been a more rubygems way to fix it.

Honestly I found the multiple versions approach more complex, confusing and more hacky. A virtualenv is just “node_modules” that also contains a Python executable.

It’s a directory - you delete and create them at will, fast, and don’t worry or care about the system Python. Having some crazy setup that patches “require” to handle concurrently installed package versions seems insane, especially if you cannot actually use them concurrently in the same Ruby process. So, segmenting them by project (aka virtualenv) seems like the best solution.

so they’ve just automated the virtual environment creation then. There’s nothing about pip that is global or not global. It unzips files from pypi into a directory. Python, not pip, doesn’t really support a “node_modules” style setup. We use virtual environments (venv/) which is somewhat similar.

Sort of. The equivalent of "virtualenv" would be more "rvm gemsets", which is what people did before bundler. Bundler is doing something different.

Bundler is not a "node_modules" style setup. It does not require dependencies to be in a local path (although they can be, the default is they live in a system-wide location, and this does not limit functionality). It also does not support more than one version of a dependency in the same execution environment (as node_modules does) -- that really would be impossible in ruby too.

It's possible something about python's design would make the bundler approach impossible, I don't know. But it's not "dependencies are installed globally" alone, as that's true of ruby too.

We would probably all benefit in understanding better how these things are handled in other environments. And I include myself here. I think ruby's bundler really set a new standard for best practices here, and many subsequent managers (like cargo) were heavily influenced by it, although many don't realize it. But meanwhile many don't even realize what they are missing or what's possible.

Like the basic idea of having a specification of top-level dependencies (including allowable ranges) separate from a "lockfile" of exact versions in use of ALL dependencies... is just so hugely useful I never want to do without it, and I think is compatible with just about any architecture, and yet somehow JS is still only slowly catching on to it.

Not quite true. Python import mechanics are quite hackable and controllable programmatically and externally via controlling the relevant PATH env variables. It's just that no one bothers with it and instead seems to rely on the set of global folder-path lookup mechanics that are standard.

But nothing stops you from masking global-libraries with local library versions (similar to node_modules). Why hasn't anyone done this you may ask? I don't know the answer to that.

There are a few reasons you can’t have multiple versions of the same module at the same time. Consider a simple enum class defined in a package: two versions now have two different enum objects which may not compare equally. Maybe a function from module A would return Enum.A and pass it to module B, which would then compare A.Enum.A to B.Enum.A, which fails. Super confusing.

So yes, Python’s import system is dynamic enough to do crazy things, but I don’t see how we can ever retrofit that into Python.

Regarding dependencies installed into a project-local directory (node_modules): that’s a virtual environment. Just more flexible.