What people casually think of as "Python" is really a huge dynamic ecosystem of packages. Imagine that there are 60k packages, each with 1k lines of code... that's a 60 million line codebase, and it can't be checked for breaking changes. Short of continually testing your code against the latest versions of packages, you're going to hit some bumps if you haul an old code out of the vault and try to fire it up on a new system.

I don't know how Javascript developers handle this.

I handle it by running Python inside an isolated environment -- WinPython does this for me -- and occasionally having to fix something if a new version of a package causes a breaking change.

The drawback of my method is deployment -- there is no small nugget of code that I can confidently share with someone. They have to install their own environment for running my stuff, or take their chances, which usually ends badly.

Most package managers have lockfiles that allow for some degree of determinism. Of course if a library introduces breaking changes you're going to have to rewrite, but only when explicitly upgrading the dependency.

That said, I 100% agree. I don't have the answer, except that I wish that there was one official answer, for developers and deployment that was easy to explain to beginners.

For what it's worth I've been using pipenv for over a year and it works good enough. I think npm's approach is better, but not perfect. I've heard good things about yarn. I know CPAN is the grandfather of all of them. I've barely used gems but they seem like magic, and Go get your repo URI out of my code please and thank you. :-) All fun aside, what languages have it right? and is there maybe a way to come up with a unified language dependency manger?

Technically, OP didn’t have to use pipenv for anything except knowing which versions of each dependency to install (Pipfile.lock) with good old pip. Those other tools are mere conveniences. Giving up on a language for that...that’s drastic.

For interpreter/compiler version management you can use asdf [0]. It works for all popular programming languages. You can use it to replace tools such as pyenv, nvm, gvm, etc.

[0]: https://asdf-vm.com/#/

Everything is pinned to exact source revisions. You can be relatively sure to be able to git clone and nix-shell and be off to the races.

You can even go the extra mile and provide working editor integration in your nix-shell (especially easy with emacs). So you can enable anyone to git clone, nix-shell, and open a working editor.

The biggest downside is becoming proficient in Nix isn't easy or even straightforward.

While you're here, how do you do this with nix-pkgs?

I looked into using nixos for nodejs deployments recently, and was amazed to find that the versions of node in the nix-pkgs repo are just pinned to X.Y.0 releases, with no discernable way to update to a bugfix release after .0 , so... I don't see how this could possibly be used for production deployments?

https://nixos.org/nixos/packages.html?channel=nixpkgs-unstab...

What I really want is to be able to tell nix to give me nodejs at version 10.16.3 and have it do the right thing every time.

I'm happy to be wrong on any of this.

I think this is a reasonable choice, considering that the main purpose of nixpkgs is to provide packages for the NixOS distribution. It's impossible to actively maintain every single version of every package.

Closest I found was to pin the nix-channel entirely, but not different components to different versions.

    let pinnedPkgs = import (builtins.fetchTarball {...}) {};
        node = pinnedPkgs.nodejs-10_x;

https://nixos.wiki/wiki/FAQ/Pinning_Nixpkgs

I've mostly stuck to nix-shell -p foo bar baz to setup my environments. It's holding me back and I should pull the trigger on a shell.nix file, just gotta learn the language.

Occasionally I still run into weird non-deterministic issues with builds, though: https://github.com/NixOS/nixpkgs/issues/71178

You'd think "pinning" a python version + channel would avoid this.

$ nix run "(import <nixpkgs> {}).python37.withPackages([ pandas statsmodels ])"

This would drop you into a shell with Python 3.7, pandas, and statsmodels installed.

For more complex use cases, you should specify your dependencies in a shell.nix file or write your own nix package.

In Python it's a mess. Some packages specify their deps in setup.py; some in a requirements file, which may or may not be read in by their setup.py. It's not rare to have to have multiple 'pip install' commands to get a working environment, especially when installing fairly boutique extensions to frameworks like Django.

There just isn't a holistic, opinionated approach to specifying dependencies, especially at a project level. Which leaves unexpected upgrades of dependencies (occasionally leading to regressions) as a reality for Python devs.

One is pipenv [0], which works similar to yarn & co. It uses Pipfile/Pipfile.lock to define and lock dependencies. Pipenv has a major flaw: It can't be used to publish packages on pypi.org (you still need twine & setup.py for that). It's also known for being slow and somewhat buggy. Despite all that pipenv is an "official" tool maintained by the "Python Packaging Authority".

The other one is poetry [1], which works exactly like yarn & co. It uses "pyproject.toml" to specify dependencies and "poetry.lock" to lock them. Poetry does most of the things "right", but it's still an underdog compared to pipenv.

Both tools have not yet fully matured, thus there are a lot of complaints.

[0]: https://github.com/pypa/pipenv

[1]: https://github.com/sdispater/poetry

Now try that with JavaScript or Rust. If you don't fix versions, come back three months later and compatibility is usually fucked up beyond all recognition.

Some languages embraced better dependency locking because they absolutely couldn't not solve the problem.

However in SemVer a major version number change is how breaking changes are documented, so seeing a v1 to v2 change coupled with having to do some work to fix breakage is just expected, something that may well be necessary for a project to make progress.

My very brief exposure to TF seems to suggest that dev environments surrounding TF are way harder to set up than my “list of bare packages in a requirements.txt file” scenario which is sufficient for a lot of more traditional endeavors.

pipenv generates a Pipfile.lock - if it can, I've used it primarily for Airflow, and some packages within Airflow have incompatible version ranges for the same dependency, which means it can't generate the lock file.

npm sucks and all, but at least it just works and doesn't get in my way as much.

used many package managers: pip, gem, go, npm, composer, etc... npm is the only one i have recent memories of having to kill & the only one that makes the fan go off (well okay most c/c++ do that too...)

quite frankly surprised by what i am seeing about python here. i have never been into the new stuff, pipenv, poetry, pipx, etc... maybe that's where the bad experience is coming from? i even don't know when and why it got so complex...

I imagine if you're deploying docker, you probably should be developing in docker (e.g. using PyCharm's remote interpreter/docker interpreter integration).

Coming from the scientific computing realm, I've only ever done "real" code in Python until a couple of weeks ago when I was forced to do some work in Typescript. Once I got over the initial worries about the best method to install npm to avoid Python-esque problems and gave nvm a try, I was very pleasantly surprised by the package management process and was able to just get on with the work.

I've tried various Python env management solutions in the past (mostly leaning towards conda), but had recently settled on just using separate LXC/LXD containers for each project.

Like any dictatorship, that can be fine if those in charge are benevolent and competent. For programming languages, the first is almost always true, but the second is far from guaranteed. Skill at programming language development has no bearing on skill at developing a build, packaging, and distribution system.

Go is a prime example of this. The language is pleasant to use with few ugly surprises, but their build system has been awful for a decade, only now reaching a semi-decent state with modules (which are still pretty damn ugly).

With python, on the other hand, there's competition in this space, and as a result the tools are pretty nice, albeit fragmented.

But then there's rust, which has a nice language AND a nice build system. You take a big risk when building both the language and build system; sometimes it works, sometimes it doesn't. And you risk fragmentation if you don't. It's a tough choice.

python3 -m venv /tmp/foo

/tmp/foo/bin/pip -U pip wheel

/tmp/foo/bin/pip -r requirements.txt

I understand some might not like it, but really, it's simple and it works.

For individual people perhaps. For companies with older projects, newer projects, greenfield stuff, etc, no.

It's not as nice as just doing a regular pip -r, but it works and isn't that much effort.

Tkinter and mpi4py are the most recent ones I've had this problem with. I expect someone will tell me "it's trivial to install these in a venv, just do X", but X is not obvious to me.

Having said that: you generally want this packages integrated with your system (which provides a self-consistent Tcl as well as MPI-environment)

It’s super useful for maintaining static versions of python, like 2.7, 3.6 and 3.7 when you have many projects that have different python requirements.

For example I've had 3.7 on my macOS system for a while, installed from the official installer, not through Homebrew. I just installed 3.8, which pointed my python3 to 3.8. But my 3.7 was still there; I created an alias python3.7 to point to that. So I can run or install anything I want against 3.7 or 3.8.

Why do we still need pyenv? I'm not asking that antagonistically, I really don't understand at this point and I'm wondering if I'm missing something.

Specifically, pyenv will dynamically and programmatically link the $PATH for the Python executables (python, pip, etc.) to the desired version as defined either by an environment variable or by the contents of a .python-version file in CWD or an ancestor thereof. The .python-version file can be checked into version control.

I like being able to specify the global and local versions for my projects and the system as a whole. I also use it as a virtualenv manager. It works well with pipenv (which I still use in anger) and vscode.

If you're working on closed-source code or have tight control of your environment, it's enough to develop and run on a single version, rendering pyenv and whatnot unnecessary.

This post reads most to me as fad. It is an opinion, you would most likely bail out.

Very few issues arise from our choice of build tooling. Not enough to consider switching at the moment. I suspect I'll try pyenv next time I have a new dev maching to set up but only as it seems fairly painless to switch from virtualenv.

Pip works, virtualenv works. They might not be great tools, but they do the job.

I don't want to worry to much about my environment, that's why I'm skeptical about new tools. Because they might break when you least expect

Virtualenv solves 90% of the problem, and it is not like you need to revisit your setup on a daily basis.

I would be more willing to spend time on my test setup than anything else, because that affects the development experience.

It just shows that Python is used for a lot of purposes and there is no single tool that handles all usecases.

> I gave up and started rewriting the project (web scraper) in Node.js with Puppeteer.

And when you'll get to the point where you need to work with different node projects, you will also need several tools to manage node versions and different environments, so that doesn't help at all.

* "I need to develop against multiple Python versions - various Python 3 versions (3.6, 3.7, 3.8, mostly), PyPy, and occasionally Python 2.7 (less and less often, thankfully)."

* "I work on many projects simultaneously, each with different sets of dependencies, so some sort of virtual environment or isolation is critical."

So sure, there is more than one way to manage a Ruby runtime version, but are there any competitors to Ruby's Bundler? I feel like it's the undisputed champion of Ruby dependency management, working with Rubygems, another unopposed incumbent in its own space, and it never even occurred to me that my language should have more than one of either such tool. Can someone help me understand what drove the Python ecosystem to need more than one dependency manager, ELI5?

I am pretty cloistered as a Ruby developer, but my limited professional experience with other language runtimes tells me that the Ruby "developer environment" experience is really second to none, (change my mind.) Is there any tool which is nearly comparable to Ruby's Pry for Python or other interpreted languages? (I doubt that there is!)

Managing gemsets and multiple versions of Ruby is old-hat.

These days, most productions environments are effectively isolated containers. If your production environment is a container, you probably should develop in a container as well. In that case you don't need much tooling for isolating an application's dependencies from other applications.

The tooling that you need is to build a python application, which means (1) get the dependencies (2) copy over some source code (or invoke the C-compiler if build a C<->Python extension) (3) run tests. Python's builtin setuptools does that fine. It didn't strike me as amazingly simple, but it's not amazingly complex either. pip is essentially a convenience wrapper for setuptools, i.e. pip is to setuptools as apt is to dpkg.

Basically, I believe that because of docker isolation is a irrelevant criterion by which to judge a language/ecosystem.

I'm a developer that supports multiple production applications, and I frequently need to make my development environment as similar to production as possible in order to reproduce a production issue for debugging purposes. I depend on that isolation to be able to do this. My work environment is such that I'm not generally permitted to use containers in production (yet). So it stands to reason through your argument that perhaps I shouldn't use them in development either. It sounds like if I were using Python as well instead of Ruby, I'd be having a much harder time.

Honestly, if I could use containers in dev and prod I would, I truly do believe the grass is greener ;) but I would not sacrifice this marvelous isolation tech, in fact I'd prefer to take RVM with me into docker-land so that I can A-B test Ruby versions within the same container image, and be guaranteed that all my cluster's worker nodes will not have to take extraordinary measures and carry both images in order to ensure the application can boot without a download delay, whenever we have to revert the canary (or whatever other minor potentially reversible lifecycle event would normally trigger a node to need to download a new, expensive base set of image layers all over again.)

This works really well: https://github.com/ms-ati/docker-rvm

poetry == npm

pyenv == nvm

pipx == npx

No big difference, IMO.

Package Management - npm? bower? yarn? Which should I use this week?

Interpreter Versions - Revisiting a project I last touched 2 years ago on node 8 has a load of locked dependencies that only work on that node version. OK, let's bring in nvm so I can get a dev environment going.

Executable Packages - oh no I've got two different projects with different and incompatible versions of apollo, what do I do? Oh right, sure npx for executable isolation so we don't pollute the global namespace.

Every ecosystem has these problems, and if they don't it's probably because they're still relatively esoteric.

Exactly! I'm not aware of any non-compiled language where (all) these issues are solved much better. I can be very productive with the tools I mentioned above and I'm glad that they work almost identical for both my main drivers (Python and JS/TS).

Since Python did not prescribe a way to handle it the community has invented multiple competing ways to solve the problem, most of which have shortcomings in one way or another.

To further add to the confusion, most Linux and Unix-based operating systems (Linux, MacOS, etc.) have their own system Python which can easily get fouled up if one is not careful.

This is one place where Java's use of a virtual machine REALLY shines. You can build an uberjar of your application and throw it at a JVM and (barring JNI or non-standard database drivers) it just works. There is also usually no "system Java", so there is nothing to break along those lines.

Exactly. It's not Python's only problem, but far and away the most painful snags I've hit with packages is when they use C code, and thereby drag in the whole system. "I'll just `pip install` this- Oh, I need to install foo-dev? Okay, `apt-install foo-dev`... oh, that's in Ubuntu but not Debian? Well this is gonna be fun..." Now I trend a bit more exotic in my systems (NixOS, Termux, *BSD, ...) but if my Python were just Python it would just work so long as I have a working Python install; in practice that's rarely enough.

Oh, but there is :( `jenv shell 1.8` is muscle memory for me now.

But in any case, it just becomes one more version of Python to consider. If you're already dealing with multiple versions, what difference does it make?

If you have root privileges and you run "sudo pip" commands you might accidentally break the specific Python dependencies that the system scripts rely on. See https://developers.redhat.com/blog/2018/11/14/python-in-rhel...

There's no issue with using the system Python, but any Python packages should be installed via yum or similar Red Hat / Fedora tools and not pip.

Note that the newer versions of RHEL have created developer-specific tool packages to separate the system packages from developer packages. This allows the developer packages to get upgraded quickly so developers have nee, shiny tools without breaking the compatibility that the base system needs to keep running.

Given that: are you sure problems arise around pure python-packages (which generally have a well enough forward compatibility), or is the problem with all the cool "machine-code"-embedded packages (which are a lot!)? And yeah, these might indeed break randomly, when installed on different systems of different time-periods. But that's a problem all binary packages have!

There is a lot of tools and confusion between versions especially because of the breaking changes between v2 and v3 (try to explain someone why he has both and when he types python it will take v2.7 as default).

I love the elegance and simplicity of the language and many tools written with it but this is a point I'd really much appreciate to be improved.

Because of that I sometimes just rewrite something before fixing it with 2.7. It's perhaps a bit more work sometimes but not as frustrating as trying to get something running that is deprecated in parts.

In Smalltalk you change parts of the environment. In Python, Java (and Ruby?), you change the entire environment, as described in TFA.

https://www.quora.com/What-is-the-essence-of-Smalltalk

    ./gradlew run

If Python would have a similar concept rather than depending on a global module location we would be able to replicate the same developer ergonomics as we have for the JVM.

Haskell with new-style cabal works like a charm with `cabal init` for package setup. And then ghci from there will give you...

- expression evaluation ofc

- type-at-point via type hole annotations

- Laziness inspector via :print

- Breakpoint debugger (someone just posted a nice Reddit text post about it today)

- Package information of terms via :info (and :load $MODULE to get any top-level term in $MODULE into repl scope)

- Docs via :doc (this is pretty new)

- Module export lists via :browse

off the top of my head

  $ curl -sSL https://get.haskellstack.org/ | sh
  $ stack install turtle

  $ cat >> ./hello.hs <<EOL
  #!/usr/bin/env stack
  -- stack --resolver lts-14.14 script

  {-# LANGUAGE OverloadedStrings #-}

  import Turtle

  main = echo "Hello, world!"
  EOL

  $ chmod u+x ./hello.hs
  $ ./hello.hs
  Hello, world!

On the contrary, I seldom had these kind of issues with projects coded in C#, C or C++: most of the time the few steps to compile the project succeeded and produced a usable binary.

As far as virtual enviroments go, I actually kind of like them. They were containers before containers became a thing, and I’ve had much fewer issues with them than say NPM, but they are more burdensome than adding full binaries to your C# project.

Where compiled languages shine, and maybe I’m misunderstanding you, is when you need to distribute and executional to end-usere. C# is much better at that than Python, but we haven’t actually done that at my shop, in around 10 years.

That said, once you get it down, you're not burdened by it much/at all. You can start and begin work on a new project in seconds, which feels important for a language that prides itself on ease of use.

But yeah, not a great look for newcomers.