People keep saying that about every new solution. But then another one comes along that's even better-er, and the previous one peters out.

The biggest need for a package manager and its ecosystem is continuity: the stance that new features and paradigms will be gradually shifted toward — without package-ecosystem incompatibilities, without CLI commands just disappearing (but instead, with long deprecation timelines), etc.

In other words, an officially-blessed package manager is one where, when something better-er comes along, it gets absorbed by the existing thing, instead of replacing it.

That is what the Python ecosystem is missing.

I don’t think it’s that another one comes along that’s better as much as it is the new “better” ends up missing some important corner case. Pipenv advertised itself as solving all problems, but once someone tries it in practice, they realize that it introduces a new problem: every little interaction takes literally 30 minutes for any non-toy project. I’ve heard mixed things about poetry, but I wouldn’t be surprised in the least of it failed to behave as advertised just because this has been my experience with every package manager I’ve tried to use. And it’s embarrassing when every other language has a package manager that just works.

EDIT: It was probably misleading to characterize Pipenv as advertising itself as solving all problems; it’s probably more correct to say that it’s significant weaknesses weren’t advertised and thus one has to invest considerably before discovering them for oneself.

Just a heads up to anyone who hasn’t looked recently: pipenv has been very actively worked on since earlier this year and has had four updates that fix a lot of issues. Earlier this year I would have said Poetry is better hands down, but after the updates and after using poetry and seeing some of its quirks, it’s a much closer matchup.

If it wasn't so opinionated it might have been more successful.

Just one example: you want your virtualenvs to be created in ~/.virtualenvs so that pipenv is a drop-in replacement for virtualenvwrapper+pip? Tough luck for you, Kenneth Reitz doesn't think that's how it should be done.

At least 3 or 4 times some issue I've wanted resolved I found in the issue tracker with the last message "we'll have to check with kennethreitz42 whether we're allowed to change that" and then silence for a year.

It could still catch up with poetry, but from what I've seen there's a fundamental mindset difference in how change requests are approached between pipenv and poetry.

Last I checked (3-4 month ago) Pipenv only cares about the situation when you are deploying code on machines (or containers) you have complete control over. If you're writing code for deploying on machines you don't have control over, via for example pip install, then pipenv isn't helpful while poetry supports this out of the box.

Interesting. I haven't used pipenv on any very large projects, but I'm surprised to hear about the slowness. With the (admittedly small) projects I've tried it, I found that it does more or less just work.

As I understand it, the problem is that Pipenv needs to resolve the dependency tree to do just about anything; however, the dependency tree is dynamic—to determine a package’s dependencies, you have to download the package and run it’s setup.py. To get the whole tree, you have to recursively download and run each package. So the cost is proportional to the size of the dependency tree, so it’s very plausible that it works fine for the smallest projects.

> The biggest need for a package manager and its ecosystem is continuity: the stance that new features and paradigms will be gradually shifted toward — without package-ecosystem incompatibilities, without CLI commands just disappearing (but instead, with long deprecation timelines), etc.

I disagree. I used to think that that's the problem, but having seen a few more cycles of it, the problem isn't that kind of commitment - after all, the whole python ecosystem enthusiastically jumps into the new thing, and Python people are used to relatively short deprecation cycles. The problems are the actual problems; every Python package manager is just embarrassingly awfully bad as soon as you try to use it for 5 minutes, presumably because they're developed by Python people who've never used a decent package manager and so think that no-one could ever need deterministic dependency resolution, once you've pinned a transitive dependency there surely wouldn't be any reason to ever want to unpin it, having the package manager coupled to the language version is absolutely fine, no-one could ever want a standard way to run tests ...

What the Python ecosystem actually needs is a single opinionated perspective on versioning that is followed by everyone, such as NPM's semantic versioning. In the absence of that I don't see how dependency resolution and thus packaging is ever going to improve in Python.

Guido is very opinionated...

He just doesn't care about package management.

Yes, Poetry should be the blessed package manager.

I only want Poetry to become the be-all-end-all of package managers if it turns out that Python really is never going to fix the core problems that have engendered so many of the hacks upon which Poetry (and its competitors) is precariously balanced. Pyenv and venv, for example.

If you were doing a green field redesign, how would you want Python to fix the core problems?

> Yes, Poetry should be the blessed package manager.

Last time I tried to use poetry (and this is why it was the last time I tried to use poetry), it ignored global pip settings and had no documented mechanism for its own settings (I believe poetry uses its own implementation of or captive install of pip) which made it completely unusable in a corporate environment with annoying SSL interception issues to work around where pip + venv worked.

Poetry is a much smoother experience when it works, though.

It will install a virtual env if you don't have one active and it will use the active one if you do. What global pip settings, for example?

Although I generally like it, the two major issues I have with poetry are the abysmal dependency resolution times and the handling of binary wheels.

> People keep saying that about every new solution. But then another one comes along that's even better-er, and the previous one peters out.

I think this is happening these days frequently. People try to cover all use cases and then end up in biting more than they can chew. It won't work that way. Good set of MINIMALS, is easy to maintain, sustain and extend.

Much of Python's growth has been driven by data science. Here, the conda package manager is pretty ubiquitous. Conda packages system and other non-Python dependencies (such as the CUDA SDK), removing the need for data scientists to resolve these non-trivial dependencies themselves. This is likely unneeded/ unwanted for production web app deployments.

Given the varied use cases for Python, the goal of a single package manager may be misguided.

My understanding is that the people who developed Conda would love to have stuck with pip, and originally wanted to see about upgrading pip to support their use cases. And it was GvR himself who told them that that wasn't going to happen.

That was a long time ago, though, when scientific computing was a small niche for Python. It might have been reasonable to say it's not worthwhile to take on all that extra work just to support the needs of a small minority of users. Fast forward the better part of a decade, and it turns out that scientific computing did not stay a small niche. I think that one could make a strong argument that, in retrospect, that brush-off did not end up ultimately serving the best interests of the Python community. It made the community more fragmentary, in a way that divided, and therefore hindered, efforts at addressing what has proven to be one of Python's biggest pain points.

Conda predates pip by perhaps a decade.

Really? The first release of pip was in 2011 [1] and the earliest release of Conda I can find is 1.1.0 in Nov. 2012 [2], and the first public commit (into an empty repo) was a month earlier [3].

[1] https://en.wikipedia.org/wiki/Pip_(package_manager)

[2] https://github.com/conda/conda/tags?after=1.3.0

[3] https://github.com/conda/conda/commit/c9aea053d8619e1754b24b...

May be anaconda that I'm thinking of.

Anaconda was released in 2012, as well. Conda is a tool that is part of anaconda.

This one: https://en.wikipedia.org/wiki/Anaconda_(installer)

I forgive myself, it's pretty confusing :D.

Ah. Name collisions suck.

Especially when they conceptually do the same thing.

And that's another issue with Python ecosystem

Why do you think Python is more susceptible than other platforms?

It is true that PyPi was designed before the author/project naming scheme popularized by github. Other than that I don't see a greater problem with name collisions in Python.

Susceptible - yes, all platforms could fall to this.

That's why a strong leadership in a community, or subcommunities, works well. Python lacked this leadership, that leads to millions of half-arsed projects that compete... without moving the whole platform forward. It feels like NIH syndrome has permeated Python. Hopefully that's going to change

I also use poetry for everything. I have 0 problems, things work on my mac, my interns pc, aws instances, I don't even see what problem people are having. Before that I was using pipenv, and before that just good old requirements.txt - there were a few occasional issues, but really not much even then. At this point, I suspect it is more about regurgitating a complaint than a real issue. But, I could be lucky and completely wrong...

- until a few months ago no way to sync an environment with a lockfile (remove packages that shouldn't be there)

- no way to check if the lock file is up to date with the toml file

- no way to install packages from source if the version number is calculated (this will likely never be fixed as it's a design decision to use static package metadata insetad of setup.py, but is an incompatibility with pip)

- no way to have handle multiple environments: you get dependencies and dev-dependencies and that's it. You can fake it with extras, but it's a hack

- if you upgrade to a new python minor version you also have to upgrade to the latest poetry version or things just fail (Something to do with the correct selection of vendored dependencies. May have since been fixed -- new python versions don't come out all that often for me to run into it. And in fairness the latest pip is typically bundled with each python so it avoids that issue)

I still use poetry because it's more standard than hand-rolled pip freeze wrapper scripts, and there's definitely progress (the inability to sync packages was a hard requirement for me but is not fixed) but it's not quite there yet

Interesting, i usually rebuild my env from pacakges so don't notice 1,2, or 3. I guess 2 should be fixable by poetry by including more from the toml in the lock file. Point 4 also didn't bother me as I in general just have the main and dev deps, this seems an easier thing to fix for poetry though. I actually have encountered 5 when fiddling around with pyenv.

If you don't need c or c++ dependencies it's ok. If you do, it's very very painful. To be fair, most of the DS libraries can be handled by conda, but if you need both conda and pip, then you're going to have a bad time. (Source: this is my life right now).

Oh man, this is my life right now, too. In my case, we're using tensorflow or tensorflow-gpu, depending on the host system and, unfortunately, only Conda offers tensorflow-gpu with built-in CUDA. Add to this that the tensorflow packages themselves are notoriously bad at specifying dependencies and that different versions of tensorflow(-gpu) are available on conda-forge, depending on your OS.

Tensorflow is the worst (along with ReAgent from FB).

I think it's because they have their own internal build systems, but they never play well with pip/conda et al.

One of my recent breakages was installing the recsim package, which pulled in tensorflow and broke my entire app. There's actually a recsim-no-tf package on PyPi, presumably because this happens to loads of people.

I see, I miss a lot of issues as don't use any GPU stuff, mainly flask + scipy and friends.... probably this is what saves me.

It's not even the GPU versions, even the CPU stuff causes issues.

The core problem is that pip will happily overwrite your existing dependencies when you attempt to install a new package.

If you don't know what the problems with pipenv or requirements.txt were, you're really not qualified to judge whether poetry has solved them or not.

You are reading wrong, it did solve my issues with reqs and pipenv. Also you certainly aren’t qualified to judge my qualifications.

Yes, Poetry is great! I avoided Python for a long time due to it's bad package management/environment handling situation, but Poetry solved all my problems there.

I wouldn't advise using anything other than the tools blessed by the PSF for mission-critical stuff. Using Poetry for local development is fine but don't build a huge infrastructure around it and don't use in production.

I migrated the CI/CD of my company to Poetry some time ago, it worked fine for some time until we needed a feature that Poetry didn't support. I submitted a PR adding the feature to Poetry but their sole developer was apparently taking some time off and the project remained without any development for several months.

I migrated the CI/CD to use my own Poetry fork but it was very cumbersome, Poetry has a very weird build system so forking it is not simple.

At this point, I realized that I was just wasting time. There is nothing that Poetry does that the other (old and stable) tools don't do. Poetry was the result of me falling for the shiny toy syndrome.