I feel like this is why things like Docker exist.

For a language like python, I'm not sure this is something you would really want to try to solve at the language level. Let's say you solved it for pure python code. You'd still have the problem of interfacing with native libraries and the OS itself. For instance, NumPy and SciPy are typically kind of a pain due to Fortran dependencies.

I think Docker is a pretty nice solution. I feel like for any language, any time this becomes painful, I assume I should either be using Docker or AWS+Chef. Disposable VMs essentially.

In a sense, virtualenv is a lot like a VM. It provides a reliable execution environment on demand without forcing you to have an actual different machine. And you don't have to deal with virtualization. But. . . it has its limitations. Because, if this is what really matters to you, you might really be wanting a VM.

Languages that run in VMs already (like Java) solve this problem a bit better at the language level. But Java dependency management turns out to be a nightmare anyway. This is probably just one of those hard problems. But if you really want reliable environments for just your app, I think it's going to be better in the long term to think about containerization and virtual machines instead of virtual environments.

You may not realise it, but Python usage patterns have changed significantly over the last 6-8 years and they are still changing. With such change, it is hard to create any tooling that is concrete and stable over the long term. By that same reason not a lot of tooling can be adopted by the standard library.

Looking at the whole thing superficially (i.e. just throwing something at the wall here), this is how I'd have done it:

* packages are by default always installed in user space.

* package manager is built into import system. Importing a package that's not in the user's environment yet will lead to something like pip install at script startup.

* import statements understand versions. By default the latest version of a package is loaded. If something else is needed, the programmer can specify the specific one.

* for most third party packages that are publicised in pypy it's recommended to bind all imports to specific versions of dependencies, so it can be tested in this configuration at release.

  > package manager is built into import system.
  > Importing a package that's not in the user's
  > environment yet will lead to something like
  > pip install at script startup

  http://lonetwin.github.io/blog/html/2016/05/16/auto_install_missing_python_modules.html

It would be very nice if the Python version was something specified in the project manifest. Don't know what is the popular build tool in python, but something like specifying the node.js version in the package.json file would be great.

I think the reason it never works like that is that the language is written before the package manager and VM/compiler.

In Clojure it's the other way around - the JVM and the package managers were there before the language. The Clojure compiler is a library that you include in your project.clj file like any other lib. Hence you can specify language version per project. That's a feature I would like to have in other programming environments.

Are you suggesting that package versions should be explicitly provided as part of import statements? Or that import statements should be mangled depending on the versions specified in setup.py?

Yep, see also my suggestion in the other subthread.