Sounds like a skill issue that shouldn’t be seen in “10k+/seat/year software.” I wonder what other fails are in it.
The Python community needs to solve this ASAP. This almost weekly pain point has turned a language I used to love in college into one of my most despised languages. The fact that the ML community uses this broken platform is infuriating.
Make a Python version 4 that focuses only on fixing the packaging. 100% per-project hermeticy. No global packages whatsoever. Solve just this issue and bring the entire ecosystem on board. Kill all the various virtualenvs, the anacondas, global packages. All of it.
Learn from Rust/Cargo. That project does it mostly right (sans lack of namespaces and reproducible builds).
I have resigned myself to repeatedly smashing my keyboard on the desk until it rains keys in blind rage induced frustration to let off steam and then calmly creating an issue in the appropriate repo instead of allowing even a modicum of hope that this will ever be fixed systemically.
That's not hard. Java's package management is well thought out and sane.
The bar is so low, you have to excavate it from a landfill of rotten node_modules...
The issue was, as you say, the introduction of ESM. It used to be that you required modules one way and one way only (yes there was AMD for advanced use cases, but it was an add-on), then people felt the need to “standardize” that, no we have this mess of ESM and CJS.
The alternative to micropackages has significant downsides. Pulling in extra surface and rolling your own buggy implementations while waiting for some commitee to bikeshed years on the implementation.
Making the right thing easy rather than the wrong thing hard is a lot better approach.
The advantage is that it's less "magic" than, say, npm. No need for rules saying how to find node_modules, you just have a PYTHONPATH env variable.
The Rust and Java approach is to do everything through a build tool's CLI, and I can't complain about that. It's probably the best compromise: Less magic than npm, and more user-friendly than Python.
There's also usually one global package dir, but that's almost exclusively used for runnable binaries, and not even really needed when there's npx.
If you think Python packaging has less magic, you probably don't know it very deeply.
I lost all hope for Python package management to ever get better when PEP 582 got rejected for some stupid tribal reasons.
Actually building Python packages is pretty complex, but that's the case for JS too. Java avoids this by distributing compiled libraries.
Traversing to the parent is especially nice for scripts. In python having scripts outside the module directory is quite painful.
Gyp is a lot easier than Python setup.py. The "easy" Java packages are comparable to pure Python/pure JS packages. With e.g. C bindings JNI/Java packaging is a horrid pain.
The python packaging system has to be used in case studies of worst design decisions of all time.