News for hackers. Stuff that matters.
Re:Uv's killer feature is pulling in local dependencies by zahlman on Sunday January 12, 2025 @09:13PM (#42679317)
(I will likely base a blog post in my packaging series off this comment later.)

What people seem to miss about Pip is that it's by design, not a package manager. It's a package installer, only. Of course it doesn't handle the environment setup for you; it's not intended for that. And of course it doesn't keep track of what you've installed, or make lock files, or update your `pyproject.toml`, or...

What it does do is offer a hideously complex set of options for installing everything under the sun, from everywhere under the sun. (And that complexity has led to long-standing, seemingly unfixable issues, and there are a lot of design decisions made that I think are questionable at best.)

Ideas like "welllll I use poetry but pyenv works or you could use conda too" are incoherent. They're for different purposes and different users, with varying bits of overlap. The reason people are unsatisfied is because any given tool might be missing one of the specific things they want, unless it's really all-in-one like Uv seems like it intends to be eventually.

But once you have a truly all-in-one tool, you notice how little of it you're using, and how big it is, and how useless it feels to have to put the tool name at the start of every command, and all the specific little things you don't like about its implementation of whatever individual parts. Not to mention the feeling of "vendor lock-in". Never mind that I didn't pay money for it; I still don't want to feel stuck with, say, your build back-end just because I'm using your lock-file updater.

In short, I don't want a "package manager".

I want a solid foundation (better than Pip) that handles installing (not managing) applications and packages, into either a specified virtual environment or a new one created (not managed, except to make it easy to determine the location, so other tools can manage it) for the purpose. In other words, something that fully covers the needs of users (making it possible to run the code), while providing only the bare minimum on top of that for developers - so that other developer tools can cooperate with that. And then I want specialized tools for all the individual things developers need to do.

The specialized tools I want for my own work all exist, and the tools others want mostly exist too. Twine uploads stuff to PyPI; `build` is a fine build front-end; Setuptools would do everything I need on the back-end (despite my annoyances with it). I don't need a lockfile-driven workflow and don't readily think in those terms. I use pytest from the command line for testing and I don't want a "package manager" nor "workflow tool" to wrap that for me. If I needed any wrapping there I could do a few lines of shell script myself. If anything, the problem with these tools is doing too much, rather than too little.

The base I want doesn't exist yet, so I've started making it myself. Pipx is a big step in the right direction, but it has some arbitrary limitations (I discuss these and some workarounds in my recent blog post https://zahlman.github.io/posts/2025/01/07/python-packaging-... ) and it's built on Pip so it inherits those faults and is that much bigger. Uv is even bigger still for the compiled binary, and I would only be using the installation parts.

Re:Uv's killer feature is pulling in local dependencies by botanical76 on Monday January 13, 2025 @02:25AM (#42680955)
Do you feel that Npm, mix, cargo went the wrong way, doing too much? It seems like their respective communities _love_ the standard tooling and all that it does. Or is Python fundamentally different?
Re:Uv's killer feature is pulling in local dependencies by zahlman on Monday January 13, 2025 @04:39AM (#42681697)
Python is fundamentally used in different ways, and in particular is much more often used in conjunction with code in other languages that has to be compiled separately and specially interfaced with. It also has a longer history, and a userbase that's interested in very different ways of using Python than the "development ecosystem" model where you explicitly create a project with the aim of adding it to the same package store where you got your dependencies from. Different users have wildly different needs, and tons of the things workflow tools like uv/Poetry/PDM/Hatch/Flit do are completely irrelevant to lots of them. Tons of users don't want to "manage dependencies"; they want to have a single environment containing every package that any of their projects ever uses (and a shoulder to cry on if a version conflict ever arises from that). Tons of users don't want to make a "project" and they especially don't want to set up their Python code as a separate, reusable thing, isolated from the data they're working on with it right now. Tons of users think they know better than some silly "Pip" tool about exactly where each of their precious .py files ought to go on your hard drive. Tons of developers want their program to look and feel like a stand-alone, independent application, that puts itself in C:\Program Files and doesn't expect users to know what Python is. People more imaginative than I could probably continue this train of thought for quite a bit.

For many of the individual tasks, there are small Unix-philosophy tools that work great. Why is typing `poetry publish` better than typing `twine upload`? (And it would be just `twine`, except that there's a `register` command that PyPI doesn't need in the first place, and a `check` command that's only to make sure that other tools did their job properly - and PyPI will do server-side checks anyway.) Why is typing `poetry run ...` better than activating the venv with the script it provided itself, and then doing the `...` part normally?

An all-in-one tool works when people agree on the scope of "all", and you can therefore offer just one of them and not get complaints.

Re:Uv's killer feature is pulling in local dependencies by imtringued on Monday January 13, 2025 @02:42AM (#42681054)
Virtualenv should have never existed in the first place. So you claiming that UV or whatever tool is doing too much, sounds to me like you're arguing based on "traditionalist" or "conservative" reasons rather than doing any technical thinking here.

Node.js's replacement for virtualenv is literally just a folder named "node_modules". Meanwhile python has an entire tool with strange ideosyncracies that you have to pay attention to otherwise pip does the wrong thing by default.

It is as if python is pretending to be a special snowflake where installing libraries into a folder is this super hyper mega overcomplicated thing that necessitates a whole dedicated tool just to manage, when in reality in other programming languages nobody is really thinking about that the fact that the libraries end up in their build folders. It just works.

So again you're pretending that this is such a big deal that it needs a whole other tool, when the problem in question is so trivial that another tool is adding mental overhead with regard to the microscopic problem at hand.

Re:Uv's killer feature is pulling in local dependencies by zahlman on Monday January 13, 2025 @04:56AM (#42681801)
Uv doing things I'm not interested in, has absolutely nothing to do with the design of virtualenvs. But virtualenvs are easy enough to work with; they absolutely don't "necessitate a whole dedicated tool just to manage" unless you could the `activate` script that they come with.

But also, Uv uses them anyway. Because that's the standard. And if the Python standard were to use a folder like node_modules, Uv would follow suit, and so would I. And Uv would still be doing package management and workflow tasks that I'm completely uninterested in, and presenting an "every command is prefixed with the tool suite name* UI that I don't like.

There was a proposal for Python to use a folder much like node_modules: see https://discuss.python.org/t/pep-582-python-local-packages-d... . It went through years of discussion and many problems with the idea were uncovered. One big issue is that installing a Python package can install more than just the importable code; the other stuff has to be put somewhere sensible.

>So again you're pretending that this is such a big deal that it needs a whole other tool

I make no such claim. The tool I want is not for managing virtual environments. It's for replacing Pip, and offering a modicum of convenience on top of that. Any time you install a package, no matter whether you use Pip or uv or anything else, you have to choose (even if implicitly) where it will be installed. Might as well offer the option of setting up a new location, as long as we have a system where setup is necessary.

Re:Uv's killer feature is pulling in local dependencies by dragonwriter on Monday January 13, 2025 @01:31PM (#42686792)
> But virtualenvs are easy enough to work with; they absolutely don't "necessitate a whole dedicated tool just to manage" unless you could the `activate` script that they come with.

venv is a dedicated tool.

virtualenv is a different dedicated tool.

Re:Uv's killer feature is pulling in local dependencies by dragonwriter on Monday January 13, 2025 @01:29PM (#42686762)
> Node.js's replacement for virtualenv is literally just a folder named "node_modules".

Node_modules doesn't support an isolated node interpreter distinct from what may be installed elsewhere on the machine. Third party tools are available that do that for node, but node_modules alone addresses a subset of the issues that venvs solve.

OTOH, its arguably undesirable that there isn't a convenient way in Python to do just what node_modules does without the extra stuff that venvs do, because there are a lot of use cases where that kind of solution would be sufficient and lower overhead.