News for hackers. Stuff that matters.
Re:Python: Please stop screwing over Linux distros by mangecoeur on Tuesday November 16, 2021 @05:34AM (#29238983)
Everyone complained, no one addressed the fact that for years the whole of python packaging was handled by like 2.5 people.

But as a rant this, like most of the ‘but just fix it’ rants, fails to acknowledge the hugely diverging needs of different users. I could not live without conda, since it’s the only sane way to get a working recent geospatial stack. Others need to run embedded environments, or portable ones, some need long term stability while others need bleeding edge packages that haven’t been released yet. Solving for a single case is straightforward; solving for all of them , not so much.

Re:Python: Please stop screwing over Linux distros by roenxi on Tuesday November 16, 2021 @06:08AM (#29239192)
> Everyone complained, no one addressed the fact that for years the whole of python packaging was handled by like 2.5 people.

If I can make an observation - it has nothing to do with the number of maintainers. The problem is deep, cultural and occupies a difficult space where it might be a bug or a feature.

The root cause here is that the Python project, and surrounding community, have little real respect for backwards compatibility. The complexity of Python setups is driven by the need to run multiple - potentially even mutually incompatible in the case of 2.x v. 3.x - versions of Python.

All languages have packaging problems, but Python is unique in my experience in the sheer number of Python installs that I need to manage simultaneously. I still have C code that works from around the time that I learned C. I'd need another Python environment installed to say the same thing about Python.

Re:Python: Please stop screwing over Linux distros by allen_lasn on Tuesday November 16, 2021 @08:45AM (#29240463)
Python 2 was supported for 20 years. Python 2.7 (largely compatible with other Python 2 versions - and including backported changes from Python 3.1 - was supported for 12 years).

The Python project went absolutely above and beyond to support users who wanted to drag out making not particularly complex changes to their codebase for over a decade.

During this same time they improved Python 3 in response to feedback and among other effects, made it less different and easier to port, from Python 2.

Many common packages supported 2.7 up to its end-of-life as well.

One of the reasons that Python is often a source of compatibility errors is that both distros and large standalone applications embraced Python in the early 2000s, became dependent on a particular version, and then refused to work with newer versions.

Python is not responsible for all the engineering decisions everyone writing in the language has ever made.

Re:Python: Please stop screwing over Linux distros by roenxi on Wednesday November 17, 2021 @02:26AM (#29250088)
Yes. That is what a culture of not supporting backwards compatibility looks like. If they valued backwards compatibility they wouldn't have to support multiple versions of Python over long stretches of time. They would be supporting 1 version that was backwards compatible with code written over 20 years ago.
Re:Python: Please stop screwing over Linux distros by mangecoeur on Tuesday November 16, 2021 @08:25AM (#29240256)
It really does have to do with the number of maintainers. A huge part of the work is not just building a package manager, but coordinating many stakeholders to all use it. That would also mean building a package manager that covers all of those use cases from day one - what's the point of trying to make people use something that doesn't meet their needs. These things are a huge amount of work. Unsurprisingly, other widely used languages have the same situation - dozens of build and packaging systems for C/C++, Java, Javascript...
Re:Python: Please stop screwing over Linux distros by danieldk on Tuesday November 16, 2021 @06:05AM (#29239160)
And yet, other languages ship package managers [1] that are widely loved within their ecosystems and cover everything from microcontrollers to server applications.

I think what it makes it more difficult in the case of Python is that it has decades of legacy to deal with. No consistent semantic versioning, packages that expect that they can modify their package path in-place (this is a nightmare for immutable systems, like NixOS or OSTree-based system like Silverblue), a wide variety of build systems that sometimes hook into make, etc. Solving this is a hard problem.

This is why an authority like PSF has to step in and say: this is how it is going to be done from here onwards.

[1] E.g. Rust's Cargo.

Re:Python: Please stop screwing over Linux distros by uranusjr on Tuesday November 16, 2021 @06:15AM (#29239256)
Rust has and “advantage” here that it’s not generally shipped with your distro’s package manager. I think my biggest problem with this article is that the distros put Python there in the first place, and all of them apply patches to make Python work how they want it. And when it doesn’t work, it’s Python’s fault…? I mean yeah Python can definitely do something to make distributing Python easier, but it can only do so much without distros’ direct involvement.

I’ll add that most of Linux distro packaging contributors are generally very nice people, understand the problem at hand, and are very open to collaboration. But sometimes you see this kind of “it’s all your fault” complaints and it’s doing exactly the opposite of helping the cause.

Re:Python: Please stop screwing over Linux distros by drran on Tuesday November 16, 2021 @07:20AM (#29239678)

  Available Packages
  Name         : rust
  Version      : 1.56.1
  Release      : 1.fc34
Re:Python: Please stop screwing over Linux distros by turminal on Tuesday November 16, 2021 @06:09AM (#29239211)
Cargo is a good example of a survivorship bias of sorts. It's so deeply integrated into rust that if you can't stomach it you just leave the ecosystem and everyone that remains likes it ;)
Re:Python: Please stop screwing over Linux distros by pdimitar on Tuesday November 16, 2021 @07:30PM (#29248076)
Hm, so what did you dislike about `cargo`?
Re:Python: Please stop screwing over Linux distros by __float on Tuesday November 16, 2021 @06:09AM (#29239208)
Is it fair to compare an interpreted language and its package manager to Rust and Cargo? Python packages ship their source (in most cases), depend on a locally installed interpreter (with semantics possibly changing by version).

Yes, Python packages often make poor assumptions about what setup.py can do (i.e., _anything_), and so you end up choosing between "tested, supported by the author, and old" or "untested, unsupported, but up to date".

Re:Python: Please stop screwing over Linux distros by wildbook on Tuesday November 16, 2021 @06:20AM (#29239289)
Rust almost does the exact same thing, so I'd say it's fair. Dependencies (crates) are grabbed in source format and compiled locally as part of the build process, and installing Rust programs through Cargo also compiles them (and their dependencies) locally.

Some crates have the same issue where build scripts rely on outside tooling being installed, but it's definitely not common to (unless you're relying on compiling C/C++ code for FFI for example, in which case it's somewhat frequent).

Re:Python: Please stop screwing over Linux distros by __float on Tuesday November 16, 2021 @06:46AM (#29239455)
I think there needs to be a distinction between fetching crates as source at _build time_ and what happens with Python. Python's "build time" will still require the source to be present on the target machine it's deployed to -- unfortunately these tools are complex because "packaging" is only half of the issue, it's also the distribution and _deployment_ which makes things messy.

Consider that, even if you want to use packages only from your application's virtualenv, the default (footgun warning!) is that Python will still use the "system" packages -- this means you may have installed Ansible or some other tool that relies on Python and many packages from the distro package manager. But your app could pick up one of those dependencies! At best, this will work fine. But in the worse cases, perhaps it subtly behaves differently or simply does not work at all.

My understanding is that Rust will, by default, statically link all of these dependencies. This, in Python, would be like a "pex" or "par" (or one of the many other options :^)), which does make the distribution aspect much simpler. (At the cost of build-time complexity, slowdowns, and occasional incompatibility.)

Re:Python: Please stop screwing over Linux distros by regularfry on Tuesday November 16, 2021 @09:42AM (#29241038)
Cargo started as a port of Ruby's Bundler. If Ruby can do it, Python certainly can.
Re:Python: Please stop screwing over Linux distros by bhaak on Tuesday November 16, 2021 @07:38PM (#29248133)
Don't underestimate the culture differences between the Ruby and the Python communities.

With Ruby, you are expected to update frequently. New ruby versions are eagerly awaited and all the major packages are updated pretty quickly.

Re:Python: Please stop screwing over Linux distros by regularfry on Wednesday November 17, 2021 @09:36AM (#29253525)
Python is just a more fractured community, for reasons that have never really been that clear to me.

But it doesn't harm the point I was getting at, which is that being a dynamic language doesn't give Python an excuse compared to Rust in itself when there are highly successful adjacent examples. Obviously there are other factors that come into play, or it would be ancient history by now.

Re:Python: Please stop screwing over Linux distros by mangecoeur on Tuesday November 16, 2021 @08:18AM (#29240177)
Worth noting that the PSF has no authority to tell people how to do their packaging.

Rust's Cargo is 20 years newer than Python and benefits from those decades of experience. It's a very different proposition to start a new system from fresh than to try to migrate a huge and diverse community towards it.

Re:Python: Please stop screwing over Linux distros by regularfry on Tuesday November 16, 2021 @09:46AM (#29241087)
This is actually where a BDFL should step in. "We're doing X for Python 5, we're using it to manage the stdlib as well as community packages, everyone get ready because I have decided."
Re:Python: Please stop screwing over Linux distros by Macha on Tuesday November 16, 2021 @07:41PM (#29248153)
They tried this with pip.

Distros ripped it out.

They added the ensurepip module so users could add it after the fact.

Distros ripped it out too.

Arguably the advantage newer languages like Rust and Go have in this regard is they don't even consider the distro use case - you're going to get static linking and you'd better like it. Whereas Python is from an older era and tries to fit in with the local customs and so gets hammered for its inconsistency

Re:Python: Please stop screwing over Linux distros by regularfry on Wednesday November 17, 2021 @03:55AM (#29250512)
Yes, and that's fine. Distro python isn't for end users.
Re:Python: Please stop screwing over Linux distros by thetwentyone on Tuesday November 16, 2021 @09:37AM (#29240985)
And Julia's Pkg! One of my primary reasons for switching away from Python to Julia was how ergonomic modern dependency management was.
Re:Python: Please stop screwing over Linux distros by komuher on Tuesday November 16, 2021 @06:09AM (#29239207)
Kinda diffrent Rust its like 3 years old when it start getting popular. Python is 13+ years old when it starts getting very popular. You cannot compare legacy to modern solution -.-
Re:Python: Please stop screwing over Linux distros by ajsnigrutin on Tuesday November 16, 2021 @06:46AM (#29239460)
Perl is 33 years old, and perl 5.x is 27 years old.

I can literally take a 20+yo book and all the examples still work. CPAN still works. I literally have 20+yo scripts still copied from server to server, from laptop to laptop, without any changes.

Re:Python: Please stop screwing over Linux distros by smallerfish on Tuesday November 16, 2021 @06:13AM (#29239233)
Maven was launched 9 years after Java got popular. Everybody was using Ant, everybody decided Ant sucked and moved to the new solution. While Gradle is the new kid on the block, it keeps the infrastructure of dependency management and is thus compatible with Maven to that extent. It can be done.
Re:Python: Please stop screwing over Linux distros by RegW on Tuesday November 16, 2021 @06:32AM (#29239373)
Ant didn't do package management.
Re:Python: Please stop screwing over Linux distros by smallerfish on Tuesday November 16, 2021 @06:45AM (#29239450)
No, but ivy was commonly used. That said, the bigger point is that the standard way to do things can be changed. For some reason, despite (or because of?) PEP, the python community seems unable to coalesce around standard ways of packaging.
Re:Python: Please stop screwing over Linux distros by RegW on Tuesday November 16, 2021 @08:20AM (#29240198)
Perhaps I have lived too long out in the provences, but Maven was my first experience of dependency management in Java. After Ant it was a no-brainer because Ant didn't do dependencies. I didn't come across references to Ivy for some years and personally I have never seen it used in the wild.

That said, I didn't actually like early Maven that much, it was pretty inflexible and often required the AntRun plugin to do something novel. However, it is still top dog everywhere I work and very much the standard way to do things.

Re:Python: Please stop screwing over Linux distros by danieldk on Tuesday November 16, 2021 @06:33AM (#29239382)
Which is exactly what my second paragraph said :). This is a common struggle for many older programming language ecosystems, e.g. I think the same is true to some extend for C and C++.

As one of the sibling commenters mentions, there are good examples that it is possible to standardize packaging better. Maven replaced IDE-driven builds and Ant for in the Java ecosystem and added proper package management. Additionally, it required that projects start conforming to standardized layouts, by taking convention over configuration and being largely declarative. I think the Maven success story lost some of its shine with Gradle, but that's another story.

Re:Python: Please stop screwing over Linux distros by WhyNotHugo on Tuesday November 16, 2021 @06:15AM (#29239254)
Which is why I'm starting to feel like Python is the new Java.
Re:Python: Please stop screwing over Linux distros by oneweekwonder on Tuesday November 16, 2021 @06:35AM (#29239400)
Strange, because python was released almost 5 years before java.
Re:Python: Please stop screwing over Linux distros by mangecoeur on Tuesday November 16, 2021 @08:33AM (#29240340)
Python is the Old Java :P
Re:Python: Please stop screwing over Linux distros by l0b0 on Tuesday November 16, 2021 @11:58PM (#29249437)
I'm not saying it solves every problem for everyone, but have you looked at Nix? I've found it to be great for portability (at least on other Linux distros), and you can easily install anything from stable nixpkgs packages to Git repos. I'm currently using it to manage OS packages and development dependencies for some of the projects I work with (a couple at Work™).