My impression is that there's lots of different niches that have developed, each with their own needs, and trying to unify them is a mess.

If you're putting something like a web app or something else that will be bundled and distributed as a unit, then you're probably best off with something like Poetry, PDM, or pip-tools - you have a lock file for deterministic dependencies, most of your dependencies will be pure python wheels, and you only really need to test things once. On the other hand, if you're developing a library, you'll need to test against multiple versions of Python, and ideally multiple versions of some of your larger dependencies, to ensure that your library will work for as many users as possible. You'll also need to be able to build and package wheels. Alternatively, you're working in data science, and your main concern is probably making sure you can install the packages you need in the environments you're going to use them - specifically, so that they work with the GPUs and other hardware you have available. And there's still the group of people writing mainly scripts for server maintenance or other tasks, who want to be able to easily install dependencies and keep those dependencies up to date for security reasons, with the minimum number of breaking changes.

Right now, there are different tools, packaging systems, etc catering to each of these groups, and so building the One Ring of Python package management is going to involve (a) solving all of these problems, and (b) convincing all these groups of people that your general solution is better than their niche-specific solution. That's certainly not easy, I don't even know if it's all that possible.

I do think that working from the ground up (i.e building the individual components like the package metadata file, or the pypackages folder experiment) seems to be working well, in that tools seem to be coalescing around these options and finding the best ways to use them, which is all work that might hopefully feed into new official tooling. But we'll see.

> you're working in data science, and your main concern is probably making sure you can install the packages you need in the environments you're going to use them

Honest question from a web developer who sometimes has to work with Python — don't containers solve exactly this?

Unfortunately no, the problem here is that you're probably going to need a lot of compiled extensions, and some of these extensions are going to be running on your GPU (especially if you're in the ML world, but also more generally if you want to take advantage of e.g. your lab's HPC cluster). PyPI can manage some of this with the wheels system (i.e. OS, architecture, Python ABI), but there's no metadata to indicate, for example, which GPU you have available. So in most cases it's possible to just precompile all the relevant variants and let people download the best one for them, or even in some cases allow people to compile everything for themselves, but there's still situations where those aren't good options.

This is why PyTorch is famously more complicated to install via newer packages managers such as Poetry, because it requires something slightly more complicated than the existing Wheel setup, and most package managers aren't designed for that. (Pip isn't designed for that either, but PyTorch has come up with workarounds for pip already.)

Containers can't solve this problem because containers are tied to the architecture of the machine they're running on, they can't abstract that away. So even if your code is running in a container, it still needs to know which architecture, OS, resources, etc it has access to.

Not exactly no - https://stackoverflow.com/questions/63960319/does-it-matter-...

You need to be running a GPU driver on the host that supports the container cuda version.

So in theory yes, in practice, weird issue occur sometimes that really suck to debug. For example why do I get NaN loss after spending 8 days on 128 GPUs with this specific set of drivers+cuda container? (Don't hold it that way, use a matching cuda version...)

Also a lot of data scientists HATE sys-admin tasks and docker falls squarely into that for many people.

The problem is that people doing data science are not developers, so instead of just using whatever is there they are reinventing a terrible version of package management.

As I see it, there is a hierarchy of packaging needs, the base levels have been solved over and over with new, better, shiny all you need tools -- while the most tricky, complicated part has been solved over and over separately with each project.

  * Pure python -- Easy, use one of the declarative ones. 
  * Python + Standalone C -- not too bad, use the build tool.
  * Python + external (potentially distro supplied) C libraries -- Using setup.py, customized,  and different for each project. 

That last one is where Pillow, the ML space, scipy, and others live, and it's painful. Pillow has a 1000 line setup.py file to find all the optional (and 2 required) dependencies and headers on it's platforms. We've also got code to build the dependencies if necessary for packaging. To port this to some standard, we'd effectively need rpm or dpkg style build infra from PyPa, to work on all the supported platforms.

I think that's also a valid view of the problem (i.e. the further away from pure Python you get, the more complex and unclear things are). But I also think that's just the view from the "library developers" perspective — if you aren't publishing a library, but using Python for some other purpose, you are going to run into issues even at points that, from your perspective, are already fairly solved.

For example, for application developers, even if they just stick to pure Python dependencies, there's still no standard lockfile format that standard Python tools can just emit and ingest. At best, you've got `pip freeze`, but you'll need to use custom tooling to update and maintain that, or switch to pip-compile or another, more full-featured package manager. To me, lockfiles really are table stakes here, but they're not at all easy to get working in Python.

> Python + Standalone C -- not too bad, use the build tool

Doesn't that generally end up using setup.py as well?

From my understanding, build ends up calling the build-backend, which defaults to setuptools.build_meta:__legacy__, which is setup.py.

I know there are other backends, but they seem very specialized to a certain project's needs.

I think there's a cmake backend too, but I don't like requiring my customers to install cmake first, and that dependency can't be expressed in pyproject.toml.

I had hoped that redo (https://redo.readthedocs.io/en/latest/) would become popular, as a small, simple, pure-Python Makefile replacement, and that there would be a back-end using it, but neither happened.

My specific needs for a backend is to support and configure a code-generation step when building my C extension. The full code generation is >10MB, which handles all 3x24 or so different specialized implementations of the core algorithm. This takes a while to compile, so during development I use a slower, general-purpose implementation.

It seems to stem from the Python packaging story being driven by PyPA, rather than the Python foundation, which appears reluctant to pick a packaging solution and ship it with CPython. setuptools was kind of an in-between, it was included in CPython but in many cases you have to update it or add plugins...

I’m thinking https://astral.sh/ are going to be the ones to “solve” this.

Unsure if slick promo or not, but that review by tiangolo instantly piqued my interest

Astral and Ruff look great, but what is their monetization strategy, and how do their interests intersect with the interests of the developer community in the long term?

Packaging and linting are two very different topics. Just because someone does A very well, doesn't mean they have any clue about B.

By joining forces with Pyflow, or a different tool, or making yet another one?

Python leadership did the python 2->3 adventure. So.....

But yeah, I agree. This is something that golang nails. They supply all the options out of the box.

I believe PEP 517 was how the Python leadership wanted to improve the situation. Unfortunately it backfired hard. It broke the existing systems and further fractured the ecosystem.

Packaging was the reason I left python (in favor of Nim) roughly 10 years ago.

Python's situation is pathetic. Some Python folks wanted to get rid of the builtin module `distutils`. Got deprecated, and now it's finally removed in Python 3.12. That's a breaking change in a minor release, probably because they're too afraid to do a Python 4 release after the 2->3 debacle. Then they publish a migration guide only as part of the 3.12 release, which is too little too late, cause the majority of the projects hasn't moved away from `distutils` yet. There's a lot of unclarity like "you can just install setuptools, it replaces distutils" (setuptools has a startup script that hacks the search paths so you can still run `import distutils` and get their vendored copy of it). Except nobody knows whether to add a conditional dependency on setuptools then. Why on earth do you have to depend on another package to get a vendored copy of distutils? And a fraction of the projects doesn't care and makes it a user problem: just install setuptools yourself as a system / user site-package, then everything works. (Except it doesn't when setuptools is in PYTHONPATH cause the startup script doesn't run, which happens on nix and other prefix-based package managers). At the end of the day, every Python package has their own ugly solution.

All that could have been prevented by the Python folks who insisted on deprecating a builtin module. They could have jumped in and help migrate the popular Python packages away from distutils long ago, to set an example. But nope, they really like to keep things messy.

You think that's the only module they dropped?

They took away cgi… which I was of course using.

And crypt.

They don't care about breaking things. People would not migrate to python4 so they just break everything in python3 :D

The cgi module is still in 3.12. It will be removed in 3.13, which is planned for release on 2024-10-01, so not yet dropped.

FWIW, the Python developers have wanted to drop the cgi module for over 20 years. It has no maintainer.

Depending on your needs, you might consider:

1) versioning and maintaining your own copy of cgi.py;

2) use something like https://pypi.org/project/legacy-cgi/ (I have not vetted it) which is a forked copy of cgi and cgitb; or

3) use the CGI capabilities of wsgiref (I used it for one CGI project to allow a future transition away from CGI).

As for "crypt" - wow! I haven't seen anyone use that since the 1990s!

Again, the code is short, and easy to vendor, except for the _crypt extension module.

For that, you might be able to use ctypes, like this:

  >>> from ctypes.util import find_library
  >>> find_library("c")
  '/usr/lib/libc.dylib'
  >>> from ctypes import cdll
  >>> libc = cdll.LoadLibrary(find_library("c"))
  >>> import ctypes
  >>> libc.crypt.argtypes = [ctypes.c_char_p, ctypes.c_char_p]
  >>> libc.crypt.restype = ctypes.c_char_p
  >>> libc.crypt(b"toomanysecrets", b"az")
  b'azi4LBG1VJohQ'

Here is what Python's own _crypt does:

  >>> import _crypt
  >>> _crypt.crypt("toomanysecrets", "az")
  'azi4LBG1VJohQ'

> FWIW, the Python developers have wanted to drop the cgi module for over 20 years. It has no maintainer.

It works. Does it need a maintainer?

The point isn't how to replace them. The point is that I have stuff that works and has been working for several years and will stop working.

Then Python is not the language for you, and never has been.

The Python core developers have had a practice of removing standard library packages for over 30 years, including packages that - like cgi.py - worked.

For example, modules removed with Python 2.0 included cmp, cmpcache, dircmp, dump, find, grep, packmail, poly, stdwin, util, whatsound, and zmod.

Python 2.4 removed mpz, rotor, and xreadlines - no more Enigma machine emulation for you, and I had to change my code because I used xreadlines.

Python 3.0 removed even more modules: cl, md5, sha, rfc822, and more. Plus it did some library reorganization.

Ever use the "parser" module? I did. It was removed in 3.10 because of the switch to the PEG parser.

PEP 594 re-affirms the reasoning behind the long-given practice removing old packages.

There are language communities with a stronger commitment to not breaking old code. COBOL is fantastically backwards compatible, for an obvious example.

I urge you to consider the options I gave. The functionality you want can be done using the standard library without that much effort.

What does it mean to say "it has no maintainer" about code in the Python standard library? In most other projects the standard library is jointly maintained by the core team. In version history the cgi module seemed to receive small fixes couple of times a year from different people.

"Has no maintainer" means there is no one who has said they have the expertise to be able to render final judgment on a bug report or other issue. "If no active maintainer is listed for a given module, then questionable changes should be discussed on the Core Development Discourse category, while any other issues can and should be decided by any committer." - https://devguide.python.org/core-developers/experts/index.ht...

In practice, what that means is if there is a bug report, like https://github.com/python/cpython/issues/71964 from 2016, then the fix may languish for years as no one in the core team is able to resolve it. You can see several people reported the bug, along with a comment from 2022 that "The cgi module is now deprecated following the acceptance of PEP 594" so will not be fixed.

The fixes I saw likely fall into the un-questionable changes that can be decided by any committer.

The funny thing in context to this is that Perl has rock solid package management and distribution since like forever (CPAN).

If the code hasn't had any updates in 15 years, it's easy to be rock solid.