> Pradyun's point about unnecessary competition rings especially true to me, and points to (IMO) a hard reality about where the ecosystem needs to go: at some point, there need to be some prescriptions about the one good tool to use for 99.9% of use cases, and with that will probably come some hurt feelings and disregarded technical opinions (including possibly mine!). But that's what needs to happen in order to produce a uniform tooling environment and UX.

Not to mention that all efforts would be focused on improving a single package manager. Even if the one that was standardized wasn't the "best" at first (by whichever metric you prefer), giving everyone an incentive to improve that one will likely make it the best within a few years.

To me the natural choice would be pip which, as OP points out, has the advantage of shipping with Python by default. We could remedy the shortcomings of pip by incorporating the most successful features from the N alternatives that exist today.

The npm/yarn fork that formed in 2016 is the closest analogy I can think of. My impression is that npm improved on its biggest weaknesses (e.g. a lock file) and has remained the default choice in that ecosystem. I suspect the same would happen with pip if it too made significant improvements.

Shipping with python is a disadvantage; any real workflow tool needs to be able to manage multiple python installations, and if you try to use a part of the python installation to do that you've got a circular dependency problem. Plus the whole "where modules go to die" thing.

Does anything speak against flipping the relationship? i.e. using pip (or something else) as the standard way to install Python.

pip is a python program.

You might want to look at micromamba as a separate tool that can install python itself, libraries, etc. with zero dependency on a python being installed. It's a very fast tool written in C++ so it's easy to install and use almost anywhere. It uses the conda ecosystem of packages.

If you have multiple versions of python installed you can just do

python3.9 -m pip

python3.10 -m pip

The standard library is the reason why python is popular to begin with.

To me, that’d be a UX nightmare. Even if I tried to, I’m not going to memorize which of my 20 checked-out Python projects are requiring which Python invocation.

IMHO it’s absolutely the toolchain’s job to manage that for me. I’d never adopt a toolchain that wouldn’t.

You do know that python is backwards compatible right?

You can just use 3.11 with everything.

Traditionally it wasn't, and even a "minor" upgrade was a significant undertaking for a large project. Maybe that's changed.

Maybe in the 2.4 -> 2.7 transitions - Python3 after about 3.6 is pretty good, you might see some minor package breakage if you're running the very latest version, but that's fairly rare.

Generally speaking, if you're on a modern OS - like recent Ubuntu LTS or MacOS you're getting 3.8 -> 3.10 and compatibility is very good in these releases.

When exactly are you defining as “traditional”? The Python point releases have generally been painless for a long time. There was a little friction in the early 2-3 era before they made it easier to write code which worked on both without translation but in general it’s been at least a decade since I’ve cared about this except for one time where a couple of libraries depended on a private API which changed in Python 2.7.9, and that was back in 2014.

https://bugs.python.org/issue22438

If my project is a library that needs to support e.g. 3.8 through 3.11, then I’m not going to use 3.11 for my development environment.

The same goes if my project depends on a library or framework that doesn’t support 3.11.

To some extend. NPM is still (even after all the improvements) the weakest of the package managers.

There’s no doubt it’s the default though.

I've heard this claim often, but only ever from people using worse package managers than NPM, so maybe I just don't know what people want in a package manager.

Things NPM has by default that are impossible to achieve in Python-land without external tools:

* Lock files. If you check in `package-lock.json` (which is always automatically generated), then everyone who clones and installs your project will install exactly the same versions, every time. In Python you need to either run pip freeze manually when you make a change (and even then, that's not quite the same thing), or you need to use third party tools with varying degrees of manual labour.

* Local installs. Unless you explicitly ask for it, NPM will always try and set up an isolated, local environment for every project you work on. I don't think this is even possible with pip alone, which is why Python also now comes with a sledgehammer of a solution: venvs.

* Consistency. `npm install` will set the project up. `npm run <TAB>` will give me a list of useful development commands. Packages are always installed to `node_modules`. Most NPM projects work in pretty much the same way, which makes it very easy to make changes to other people's projects, or get started on your own. I have very rarely seen two Python packages that work the same. Different venv names, different places where the dependencies are specified, different procedures for installing those dependencies, different ways of locking dependencies (if any were used at all), etc, etc. That's not too get started on the different options for task runners.

* Workspaces. It turns out, it's often useful to develop multiple Python packages at the same time (because they need to be consumed by different people or different systems, or just because it's convenient to split a project up sometimes). With NPM you just need to configure this once and it'll all get linked up properly by default. How do you do this in Python? I ended up leaving my last job before I managed to figure this out, which was disappointing to my sense of professional pride, but oh so valuable for my sanity. Now I work with Javascript projects and don't need to worry about things like that because it just works.

* Multiple versions of the same package can be installed at the same time. Wasteful? Possibly. Useful? Very. When a dependency makes a breaking change, it doesn't need to split the whole ecosystem in two while we wait for all of the other dependencies to update their version. This is simply impossible with Python.

And that's just a comparison with Python, which at least has a package manager. I've heard the same claims from people who develop C/C++ projects, which can surely only be a form of Stockholm syndrome...

Even if pip was a little better I still wouldn’t use it, because Poetry solves the problem so well you’d need to pull it out of my cold hands. It solves not only the install package problem, but locking, venvs, project structure and packaging in a well integrated solution with a fantastic UX.

If poetry (or similar) were renamed to Pip and it were to be included in the stdlib id obviously switch.

It doesn’t solve the single static binary problem or docs problems, but it’s much much closer.

Everytime you install a package through poetry, you're also running pip code.

You didn't even understand the point they made: the need for Pipenv and poetry would pretty much go away if pip added support for a proper lockfile and venvs. And that's the only correct choices as pip is already pythons package manager.

What's the problem with first doing the venv thing and then running pip?

Any multi step process is guaranteed to produce random assortments of workflow tools to manage those steps which will seek to replace the original process as the one true process.

If there two steps, one part of the community will insist they belong in: a makefile, bash script, python script, lambda network service, bazel, pants, scons, terraform, ansible, npm …

So we should have 1 single shell command to do literally everything?

before poetry and pipenv was a thing, pretty much everyone was recommending the virtualenvwrapper script instead of using venv directly... so yes, thats what it ultimetaly always ends up as.

One might invest 20 seconds to learn how to use it, but to each their own :)

Not what I would or wouldn’t do personally, but what happens generally in practice.

Poetry ended up being mostly a disappointment having used it for the last 18 months. First time installs were eventually an hour long to resolve, and lock files were platform specific, making using them for reproducibility almost useless. The wider python ecosystem is frankly painful to productionize even today and it really does need some focus and governance.

Are you sure the lockfiles are platform-specific?

I spent a long time migrating a project to use poetry. One of the reasons I opted for poetry over others was that the lockfile retained all of the environment markers in the packaging metadata, so that the lockfile could support multiple interpreters and interpreter versions, multiple platforms, etc.

I can't speak for Poetry directly, but knowing how Python dependency resolution works: I don't think Poetry can make lockfiles not platform-specific, since package source distributions are allowed to (and regularly do) run platform-specific code for their own dependency selection logic.

For example, your package might depend on `foo`, which in turn could sniff the host OS and select the appropriate subdependency. You'd then end up pinning that subdependency, which would be incorrect on a different host OS.

(Similarly for Python versions: a subdependency might be required on < 3.7, so re-installing from a lockfile generated from an older Python could produce a spurious runtime dependency.)

If you use C-extensions or platform specific packages like pywin32, how would.you expect them to NOT be platform specific? Would you want it to only allow pure Python packages?

I've given up on poetry. I still to this day can't get it to use the active version of python, which you'd think would be the default. Even after setting the config flag. Setting the config flag somehow made things worse, all my poetry envs started using the wrong python. I can't even force it to use the version I want with 'env use /path/to/python', I can only pass stuff like 'env use 3.9'. What the hell is 3.9? I want to pass a specific path to a version I have.

The primary (probably build and) packaging system for a software application should probably support the maximum level of metadata sufficient for downstream repackaging tools.

Metadata for the ultimate software package should probably include a sufficient number of attributes in its declarative manifest:

Package namespace and name,

Per-file paths and Checksums, and at least one cryptographic signature from the original publisher. Whether the server has signed what was uploaded is irrelevant if it and the files within don't match a publisher signature at upload time?

And then there's the permissions metadata, the ACLs and context labels to support any or all of: SELinux, AppArmor, Flatpak, OpenSnitch, etc.. Neither Python packages nor conda packages nor RPM support specifying permissions and capabilities necessary for operation of downstream packages.

You can change the resolver, but the package metadata would need to include sufficient data elements for Python packaging to be the ideal uni-language package manager imho

> some prescriptions about the one good tool to use for 99.9% of use cases

And we've always had such tool, the humble "pip". Now it even works great with pyproject.toml. Plus built-in "virtulalenv" if you need isolation.