News for hackers. Stuff that matters.
Re:Python wheel filenames have no canonical form by g15jv2dp on Wednesday June 12, 2024 @12:21PM (#40659841)
Why is Python packaging such a mess? That's a serious question. Has someone written up some recap or history piece on how it came to be this way? Any pointers?
Re:Python wheel filenames have no canonical form by woodruffw on Wednesday June 12, 2024 @12:27PM (#40659915)
The Python packaging ecosystem is 25+ years old, and was largely informally specified until ~12 years ago. Most of the complexity in Python packaging can be traced back to (1) a large degree of pre-existing permissive behavior that needed to be encoded in early standards to get standardization off the ground, and (2) the fact that Python packaging is still largely federated, with multiple independent groups performing tool development (and interpreting the standards).

(2) has historically also been a strength of Python packaging, and has been a big source of modernization efforts over the years that (IMO) have made packaging much better: things like pyproject.toml, build backends, etc.

(I think there have been a few recap talks at PyCon over the years on packaging's history. There are definitely people way more qualified than me to provide a more detailed summary :-)).

Re:Python wheel filenames have no canonical form by rvnx on Wednesday June 12, 2024 @12:30PM (#40659960)
One example, Semantic Versioning and odd/even versioning, where 3.1 is considered unstable, versus 3.2 is considered stable, versus 3.3 unstable, etc. This was standardized only in 2010, so we can't blame Python developers for not foreseeing that in 2000.
Re:Python wheel filenames have no canonical form by XCabbage on Wednesday June 12, 2024 @01:00PM (#40660339)
The odd vs even versions rule you describe, while followed by a few projects, is not part of Semantic Versioning (aka SemVer) - there's no mention of it whatsoever in the spec at https://semver.org/.

Indeed it is basically incompatible with SemVer given that SemVer has its own way of marking unstable versions (or, equivalently, pre-release versions; SemVer conflates the two concepts). Unstable/pre-release versions under SemVer are indicated by a hyphenated suffix after the patch version - e.g. 1.2.3-alpha

Re:Python wheel filenames have no canonical form by woodruffw on Wednesday June 12, 2024 @12:33PM (#40660014)
Right -- one of Python (packaging's) small ironies is that people sometimes forget how old the Python ecosystem is, in part because the ecosystem and language have managed to keep up with the times :-)

(If we use similarly-aged interpreted languages as a benchmark, I would say that Python packaging has modernized admirably in some aspects: compare CPAN, for example.)

Re:Python wheel filenames have no canonical form by rvnx on Wednesday June 12, 2024 @12:37PM (#40660055)
Oh yes, can't agree more, compared to Perl it has completely modernized and stayed relatively well organized :) It's a very pleasant platform.
Re:Python wheel filenames have no canonical form by g15jv2dp on Wednesday June 12, 2024 @01:07PM (#40660450)
What's wrong with CPAN?
Re:Python wheel filenames have no canonical form by zbentley on Wednesday June 12, 2024 @02:30PM (#40661301)
Off the top of my head:

- Minimal standardization of ways to cache pre-built artifacts for modules with binary/XS components (no widely adopted wheel equivalent).

- Extremely poor support in cpan-the-tool for multiple local::lib environments.

- Abject lack of standardization on how hermetic/isolated (or not) builds involving linking to native libraries/compiling code should be. Talking super simple stuff here: what path-related envs to inherit, when module build scripts should hardcode vs. use system-path-discoverable versions of common C toolchain components, stuff like that.

- Standardization on build tools ("backends" in Python parlance) is poor. Module::Build is "the future", but lots of stuff seems not to use it. ExtUtils::MakeMaker is ... not good, and alternatives seem to be all over the place in terms of supportedness (I've heard Dist::Zilla is good, never tried it; spent plenty of time with SWIG hassles though). This is a low bar, and Perl trips over it. To be fair, so does Python.

- No uninstallation support (because there's no standardized manifest format). Sure, cpanminus exists (mirroring the annoyance that is e.g. conda competing with pip in Python), and even then uninstallation is dicey.

- Uninstallation troubles hint at the larger issue: CPAN packages are, if you zoom out far enough, shell/perl scripts that perform arbitrary system modifications. As jsiracusa used to say when we worked together, "they're file sprayers, not packages". If you want anything more predictable (or less dangerous) than that, you're out of luck unless you follow a very narrow path which many packages and deployment environments may not be compatible with.

- TMTOWTDI doesn't help here; even simple pure-perl modules have piss poor consistency re: how they install, so answering questions like "what will installation of Some::Module make available" is harder than it needs to be because maintainers are by turns over-clever and using dated methodologies for packaging their code.

- For reproducibility, cpan-the-tool's stance on lockfiles and hashes seems to be "what the fuck is a lockfile?". Given poor CPAN-wide compliance with Semver or equivalent conventions, that makes builds that need to happen in different environments (e.g. local, docker, CI, production) needlessly complicated.

- MacOS support for many common libraries with XS/native components is very poor even by Python's (nightmarishly bad until c. 2019, when tools matured even though the advent of ARM macs made it feel like they didn't) standard.

I sincerely hope that stuff has improved in the Perl community since I last tangled with CPAN issues on the regular (around 5.18/5.20). But given the increasing number of lonely tumbleweeds drifting around CPAN and the Perl community's tendencies in general, I won't hold my breath.

Re:Python wheel filenames have no canonical form by 0xbadcafebee on Wednesday June 12, 2024 @12:45PM (#40660162)
Because the core team never took it seriously. You can let packaging solutions grow organically if your core team doesn't want to do it, but then you need to find those who start a project, help guide them, point people toward them, help it grow and improve. You can't just throw your hands up and hope somebody else solves it.

Perl has the best packaging community I've seen. Ironically I don't think they even have binary packages, but they do have a system that tests both core Perl and most of the CPAN archive on like 20+ platforms. The packaging fits the community, it's robust, and sets conventions that the rest of the community follows that improves the entire ecosystem. It has definitely evolved over time. But I don't know of anyone who has ever said "Man, Perl packaging is a huge pain in the butt, I wish someone could solve this". If there's a problem, usually there is just one solution that everyone adopts, builds on and improves (which is ironic considering Perl's motto)

Whereas with Python, if there's a problem, someone tries to solve it, but it's not perfect, so somebody makes a second solution, then a third, fourth, fifth, etc. Python never established the convention of reusing and improving existing things. Python has a forking/independent culture. So there will always be 20 different ways to do something and none of them all that great (which is ironic considering Python's principles)

Re:Python wheel filenames have no canonical form by zackproser on Wednesday June 12, 2024 @12:37PM (#40660062)
https://www.pinecone.io/blog/pain-poetry-python/

This was the write up I did last year

Re:Python wheel filenames have no canonical form by kingkongjaffa on Wednesday June 12, 2024 @12:52PM (#40660244)
Hey this is a nice write up and I'm glad to be able to follow along, I've been writing Python since around 2012. Poetry is a step in the right direction, but I would love to see it baked into the language itself somehow.
Re:Python wheel filenames have no canonical form by doctorpangloss on Wednesday June 12, 2024 @02:26PM (#40661267)
Poetry is adopted by less than 0.5% of Python repos on GitHub. Because you can’t install PyTorch with it in a way that works for more than 1 platform, and sometimes in general, its adoption among new and sophisticated projects is trending towards zero.

I have no beef with Poetry but having taken huge brain damage dealing with Python packaging as a USP for end users, the maintainers have decided to not participate in the most exciting things in the ecosystem to the detriment of the otherwise excellent and thoughtfully designed project. Ultimately people who are offering a packaging alternative must deliver a “universal install wizard” solution that includes Windows, which interests few people to work on.

Re:Python wheel filenames have no canonical form by maleldil on Wednesday June 12, 2024 @12:56PM (#40660284)
I don't think we'll ever see proper packaging integrated into the language. Our best hope is for the ecosystem to coalesce around a particular tool, or at least have standards that all the major players follow. The latter is kind of happening slowly over time.
Re:Python wheel filenames have no canonical form by amlozano on Wednesday June 12, 2024 @12:24PM (#40659876)
Python was first released in the early 90s. If you compare Python packaging to other languages of its time, its not so crazy. For comparison, C++ was released in '85.
Re:Python wheel filenames have no canonical form by g15jv2dp on Wednesday June 12, 2024 @12:29PM (#40659952)
They weren't afraid to break pretty much everything with the release of Python 3 in 2008. That's my baseline.
Re:Python wheel filenames have no canonical form by pphysch on Wednesday June 12, 2024 @01:15PM (#40660551)
And yet Python3 is more popular than ever /shrug
Re:Python wheel filenames have no canonical form by mjw1007 on Wednesday June 12, 2024 @01:19PM (#40660591)
The early days of setuptools/easy_install (around 2004) were interesting. The way I remember it, the main developer had a clear "don't ask for permission, just build what you think is best and put it out there" approach.

That avoids the problems you can get where everything gets discussed to death and nothing actually gets written, but the downside was that the project spent a long time in an uneasy situation where it didn't have complete official buy-in but was popular enough that it would have been hard to push an alternative.

There were a couple of questionable early design decisions that didn't help.

Re:Python wheel filenames have no canonical form by rvnx on Wednesday June 12, 2024 @12:26PM (#40659901)
500'000 million packages, times n versions, all of that evolving over 30 years.
Re:Python wheel filenames have no canonical form by formerly_proven on Wednesday June 12, 2024 @01:09PM (#40660489)
Most other packaging systems don't even attempt to solve the problems that e.g. wheels solve.