News for hackers. Stuff that matters.
Re:Pyenv – lets you easily switch between multiple versions ... by mwexler on Monday March 25, 2024 @10:39AM (#39816896)
So, what is "the tool for beginners fighting with python packaging problems"?

That is, the pattern seems to be that someone mentions a solution, then a zillion responses as to why it sucks.

Is there any tool or pair that sucks least for most cases and beginners? I get that every case is different, but perhaps there are some useful starting points?

Re:Pyenv – lets you easily switch between multiple versions ... by BiteCode_dev on Monday March 25, 2024 @12:59PM (#39818658)
While there are no silver bullet, for beginners, I found that "Relieving your Python packaging pain" (https://www.bitecode.dev/p/relieving-your-python-packaging-p...) is the Pareto solution. That is, the solution that has the best ratio of effort, reward, but also the lower risk of failure. It's not no risk, but I've been helping beginners for 15 years with Python, and have tried everything you can think of.

It does imply, on linux, to limit yourself to the choices of Python you can install. This constraint is, for most people, preferable than the alternative, even if it gets frustrating to our geeky soul.

Re:Pyenv – lets you easily switch between multiple versions ... by philwelch on Monday March 25, 2024 @10:57AM (#39817131)
You could just give up and resort to using Docker.
Re:Pyenv – lets you easily switch between multiple versions ... by fishpen0 on Monday March 25, 2024 @11:53AM (#39817889)
Honestly this. I've moved three companies now to docker based dev for teams on python. It's better for teams who want to customize how they in particular work without inadvertently also blocking themselves from quickly contributing to another team because that team uses a totally different python management scheme. So a team can be conda-in-docker and another team can be virtualenv-in-docker and another team can be raw-python-in-docker and that one group of weirdos can be jython or whatever other sad shit they cook up.

The common interface is they all use `docker-compose up` and have their editors hooked into the containers.

Re:Pyenv – lets you easily switch between multiple versions ... by Zetobal on Monday March 25, 2024 @12:32PM (#39818340)
this is the way.
Re:Pyenv – lets you easily switch between multiple versions ... by BiteCode_dev on Monday March 25, 2024 @01:16PM (#39818884)
Using docker is not free, you are effectively trading one type of complexity for another.

And of course you'll have to make sure people equally have good practices, since incorrectly using sudo pip install in docker and not a venv are very common.

So again, one possible solution for a certain context, but I wouldn't sell that to most people. Certainly not to beginners.

People writing their pythonanywhere website won't pop up a container, won't they ?

Re:Pyenv – lets you easily switch between multiple versions ... by hot_gril on Monday March 25, 2024 @01:42PM (#39819226)
For a beginner trying to run someone else's Python project, it is free. You run one command and it works. There's tons more complexity under the hood, but you don't have to deal with it. This is why so many Python projects do resort to Docker.

And btw, JS doesn't have this problem, they use NPM.

Re:Pyenv – lets you easily switch between multiple versions ... by dragonwriter on Monday March 25, 2024 @01:49PM (#39819304)
> For a beginner trying to run someone else's Python project, it is free.

Its absolutely, by far, the highest friction alternative on Windows for anyone that doesn't use Docker for other purposes.

Re:Pyenv – lets you easily switch between multiple versions ... by hot_gril on Monday March 25, 2024 @01:51PM (#39819328)
Is it not just install and run? The alternative is installing Python, installing pip, etc.
Re:Pyenv – lets you easily switch between multiple versions ... by dragonwriter on Monday March 25, 2024 @02:04PM (#39819455)
Installing python with the python.org installer installs the entirety of Python and its standard library and tools, including pip, venv, etc., and the py launcher which supports selecting from multiple python versions. Linux distros may split this up into different packages, but on Windows (or Mac, afaik, though the py launcher is a Windows-only feature) using the official installers is one-stop shopping.

And, no, docker isn't just install and run on Windows. Before Docker made a heavy push for paid Docker Desktop for even personal use, it was close to that.

But now the way to get a free usable docker command line is to install WSL and a Linux environment, install docker there, and then invoke docker via wsl. (Which, of course, you will not find via Docker’s own information, which will try to sell you a paid subscription.)

Re:Pyenv – lets you easily switch between multiple versions ... by hot_gril on Monday March 25, 2024 @06:20PM (#39821869)
I tried it just now on my spare Win10 desktop, and it went like this:

1. Download and install Docker desktop for Windows

2. Restart Windows (guess you don't have to do this with just Python)

3. Run Docker Desktop

4. Say "no" to signing into a Docker account

5. Wait for engine to start, which took a few minutes the first time, a bit annoying

6. Pull and run an image (I tried nginx)

It was weird being asked to log in, but the "no" button was pretty clear. I didn't feel like I was forced to use WSL to avoid paying; maybe they've backtracked from something. Unity was far more convincing that I had to pay.

Re:Pyenv – lets you easily switch between multiple versions ... by BiteCode_dev on Tuesday March 26, 2024 @05:38AM (#39825824)
- Not if you have to edit the code. Try to explain your beginner with notepad++ how to do that.

- Not if you have to deploy the code.

- Not if you have to import the code in your own project.

Basically, not if you want to do anything by running the code. For which I would suggest an installer instead.

Re:Pyenv – lets you easily switch between multiple versions ... by hot_gril on Tuesday March 26, 2024 @12:33PM (#39829795)
I don't see what editing has to do with this. Whether you're running the main.py or the Dockerfile, if you want to edit the code then you gotta use Notepad.
Re:Pyenv – lets you easily switch between multiple versions ... by SJetKaran on Monday March 25, 2024 @10:05PM (#39823584)
Even at that, there are edge cases like when a python package like Tensorflow uses a instruction set that is not available on a Arm machine.
Re:Pyenv – lets you easily switch between multiple versions ... by philwelch on Tuesday March 26, 2024 @12:52AM (#39824366)
Yeah, personally I wouldn't develop on a different ISA than what I deployed to in production.
Re:Pyenv – lets you easily switch between multiple versions ... by IshKebab on Monday March 25, 2024 @10:51AM (#39817057)
There isn't one. Python distribution and packaging is just fundamentally horribly broken.

I think his point was that you shouldn't pretend to users that just switching to pyenv is the solution.

Re:Pyenv – lets you easily switch between multiple versions ... by notatallshaw on Monday March 25, 2024 @11:31AM (#39817586)
> Python distribution and packaging is just fundamentally horribly broken

It's clearly not because most people successfully use it fine.

The problem of distribution and packaging is often a matter of user expectations vs. the actual problem.

The user expectations is that Python is a high level language and will run the same across different machines regardless of OS and Hardware.

The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting. So people can end up moving the the glue of the project first and then expect everything else to just work.

Things are getting better (e.g. https://lukeplant.me.uk/blog/posts/python-packaging-must-be-...), a lot of people have put a lot of work into standards and improving the ecosystem, but it is a hard problem that most other popular languages don't interface with nearly as much.

Re:Pyenv – lets you easily switch between multiple versions ... by Capricorn2481 on Monday March 25, 2024 @11:49AM (#39817848)
I don't disagree with what you're saying but that article seems odd. It's just a story of someone installing something once that didn't break immediately. Not only is it anecdotal, it doesn't even seem to confirm whether the pip story is getting better or if they just got lucky.
Re:Pyenv – lets you easily switch between multiple versions ... by notatallshaw on Monday March 25, 2024 @02:08PM (#39819494)
I agree, but there is no large scale study on this.

As someone who has managed Python distributions in a large company and who triages issues on the pip github issue page that my anecdotal experience is things are getting better.

The only hard statistic I can point to is the number of top packages on PyPI that offer wheels has substantially gone up, and is close to 100% in the top 500.

Re:Pyenv – lets you easily switch between multiple versions ... by hot_gril on Monday March 25, 2024 @01:49PM (#39819303)
They use it fine by using Docker. So many major Python repos on GitHub come with a Dockerfile compared to, say, NodeJS. It's unfortunate, but having dealt with Python packages before, I don't blame them.
Re:Pyenv – lets you easily switch between multiple versions ... by notatallshaw on Monday March 25, 2024 @02:13PM (#39819541)
Lots of analysts, data scientists, traders, engineers, etc., use Python third party packages successfully, and have never touched, or maybe even head of, Docker.

And yeah, in general there are significantly less NodeJS third party packages interfacing with packages that directly depend on OSes and hardware. Python has many third party packages that are older than NodeJS that depend on foreign function interfaces, win32 com apis, directly talking to graphics shaders, etc.

Re:Pyenv – lets you easily switch between multiple versions ... by hot_gril on Monday March 25, 2024 @02:16PM (#39819565)
Python packages are painful even if nothing native is involved. There are NodeJS packages that rely on native code too, difference is the packaging system is a lot simpler to use, though it's starting to get worse with the `import` vs `require` nonsense and Typescript.
Re:Pyenv – lets you easily switch between multiple versions ... by IshKebab on Monday March 25, 2024 @04:04PM (#39820686)
> It's clearly not because most people successfully use it fine.

Well, no because it's perfectly possible to successfully use a horribly broken system. I use Python "successfully", it just meant I have spend probably literal weeks of my life fighting pip and virtualenv and relative imports and finding I need flags like `--config-settings editable_mode=compat`.

> The problem of distribution and packaging is often a matter of user expectations vs. the actual problem.

Ha yes, I expect it to work reliably and simply and it doesn't!

> The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting.

That's completely irrelevant to the kind of problems I was talking about. I run into issues with Python failing to compile C libraries relatively rarely! Even compiling Python itself seems to work quite well (maybe not surprising since that's one of the only ways to get a new version on Linux).

It's all the packaging infrastructure that's a mess. Pip, virtualenv, setuptools, and also the module import system is a total disaster. You don't see questions like this for Go:

https://stackoverflow.com/questions/14132789/relative-import...

Re:Pyenv – lets you easily switch between multiple versions ... by notatallshaw on Monday March 25, 2024 @04:34PM (#39820985)
> It's all the packaging infrastructure that's a mess. Pip, virtualenv, setuptools, and also the module import system is a total disaster. You don't see questions like this for Go:

> https://stackoverflow.com/questions/14132789/relative-import...

I think you'll find the further you delve into it the less the problems are distinct, a lot of the issues with the module import system, pip, virtualenv, setuptools, etc. is because they are designed with having to support a vast range of things, from being depended on to interact with system libraries to downloading sdists and compiling arbitrary languages, etc.

Though the specific example you linked was largely solved with Python 3, there was a lot of confusion during the 2 to 3 transition because people had to support both behaviors, but most people don't have to think about Python 2 any more.

Re:Pyenv – lets you easily switch between multiple versions ... by IshKebab on Monday March 25, 2024 @04:47PM (#39821096)
> Though the specific example you linked was largely solved with Python 3

I can assure you it absolutely was not.

> I think you'll find the further you delve into it the less the problems are distinct, a lot of the issues with the module import system, pip, virtualenv, setuptools, etc. is because they are designed with having to support a vast range of things, from being depended on to interact with system libraries to downloading sdists and compiling arbitrary languages, etc.

Not really. There are plenty of systems that have to "support a vast range of things" that aren't this bad.

In my opinion it's because the core Python devs didn't particularly care about the issue, never really tried to solve it, and as a result we have 10 incompatible half-baked third party solutions.

It's similar to the situation with C/C++ - worse in some ways, better in others (at least there is a de facto package registry in Python). In some ways it's because both languages are very old and predate the idea that packaging should be easy and reliable. That's fine, but please don't pretend that it is easy and reliable now.

Re:Pyenv – lets you easily switch between multiple versions ... by notatallshaw on Tuesday April 02, 2024 @04:47PM (#39910702)
> I can assure you it absolutely was not.

The question, as posted, was a confusion about how Python 2 relative importing worked, which was indeed bad. I don't know what you think you are pointing out, you haven't said, and the question *is* about Python 2.

> Not really. There are plenty of systems that have to "support a vast range of things" that aren't this bad. > > In my opinion it's because the core Python devs didn't particularly care about the issue, never really tried to solve it, and as a result we have 10 incompatible half-baked third party solutions.

I agree that a lot of the solutions were created when there was not an understanding, or thought out design, on what would be a good packaging solution.

But these ill thought out solutions were exactly because of trying to support this wide range of situations, from working on weird OSes, to integrating with strange build systems.

However, what you seemed to have missed is there is now a first party standard on:

* How package installers (pip, poetry, PDM, etc.) should interact with package builders (setuptools, hatchling, etc.)

* How and where build configuration and project metadata should be stored (pyproject.toml)

Almost every popular Python package tool now supports these standards, meaning they all interact with each other pretty well.

Dropping all legacy configuration, and updating the standards to support edge cases is still a long road, but it is a road being travelled and things are getting better.

Re:Pyenv – lets you easily switch between multiple versions ... by hot_gril on Tuesday March 26, 2024 @04:04PM (#39832218)
The silver bullet is Docker. It's not the best solution in every situation, but it's a passable one.

For your own projects built from scratch (rather than big multi-dep projects off GitHub), there's a smaller learning curve if you go to the vanilla https://www.python.org/downloads/ , install the latest, and use the included pip to install packages. That'll probably get you very far. It's not like the older days when you needed both Py3 and Py2.

For experts working frequently in Python, tools like Pyenv can make more sense.