Re:uv downloads overtake Poetry for Wagtail users
by matrss on Monday March 17, 2025 @06:30AM
(#43386904)
IMO if you require libraries in other languages then a pure python package manager like uv, pip, poetry, whatever, is simply the wrong tool for the job. There is _some_ support for this through wheels, and I'd expect uv to support them just as much as pip does, but they feel like a hack to me.
Instead there is pixi, which is similar in concept to uv but for the conda-forge packaging ecosystem. Nix and guix are also language-agnostic package managers that can do the job.
Re:uv downloads overtake Poetry for Wagtail users
by amelius on Monday March 17, 2025 @06:42AM
(#43386995)
But for example, if I install the Python package "shapely", it will need a C package named GEOS as a shared library. How do I ensure that the version of GEOS on my system is the one shapely wants? By trial and error? And how does that work with environments, where I have different versions of packages in different places? It sounds a bit messy to me, compared to a solution where everything is managed by a single package manager.
Re:uv downloads overtake Poetry for Wagtail users
by dagw on Monday March 17, 2025 @07:19AM
(#43387272)
You are describing two different problems. Do you want a shapely package that runs on your system or do you want to compile shapely against the GEOS on your system. In case 1 it is up to the package maintainer to package and ship a version of GEOS that works with your OS, python version, and library version. If you look at the shapely page on pypi you'll see something like 40 packages for each version covering most popular permutations of OS, python version and architecture. If a pre-built package exists that works on your system, then uv will find and install it into your virtualenv and everything should just work. This does means you get a copy of the compiled libraries in each venv.
If you want to build shapely against your own version of GEOS, then you fall outside of what uv does. What it does in that case is download the all build tool(s) specified by shapely (setuptools and cython in this case) and then hands over control to that tool to handle the actual compiling and building of the library. It that case it is up to the creator of the library to make sure the build is correctly defined and up to you to make sure all the necessary compilers and header etc. are set up correctly.
Re:uv downloads overtake Poetry for Wagtail users
by amelius on Monday March 17, 2025 @07:31AM
(#43387351)
In the first case, how does the package maintainer know which version of libc to use? It should use the one that my system uses (because I might also use other libraries that are provided by my system).
Re:uv downloads overtake Poetry for Wagtail users
by dagw on Monday March 17, 2025 @07:51AM
(#43387484)
The libc version(s) to use when creating python packages is standardised and documented in a PEP, including how to name the resulting package to describe the libc version. Your local python version knows which libc version it was compiled against and reports that when trying to install a binary package. If no compatible version is found, it tries to build from source. If you are doing something 'weird' that breaks this, you can always use the --no-binary flag to force a local build from source.
Re:uv downloads overtake Poetry for Wagtail users
by stabbles on Monday March 17, 2025 @06:52AM
(#43387072)
You could use a package manager that packages C, C++, Fortran and Python packages, such as Spack: here's the py-shapely recipe [1] and here is geos [2]. Probably nix does similar.
[1]: https://github.com/spack/spack/blob/develop/var/spack/repos/... [2]: https://github.com/spack/spack/blob/develop/var/spack/repos/...
Re:uv downloads overtake Poetry for Wagtail users
by matrss on Monday March 17, 2025 @06:54AM
(#43387088)
That's what I mean, in this case pip, uv, etc. are the wrong tool to use. You could e.g. use pixi and install all python and non-python dependencies through that, the conda-forge package of shapely will pull in geos as a dependency. Pixi also interoperates with uv as a library to be able to combine PyPI and conda-forge packages using one tool.
But conda-forge packages (just like PyPI packages, or anything that does install-time dependency resolution really) are untestable by design, so if you care for reliably tested packages you can take a look at nix or guix and install everything through that. The tradeoff with those is that they usually have less libraries available, and often only in one version (since every version has to be tested with every possible version of its dependencies, including transitive ones and the interpreter).
All of these tools have a concept similar to environments, so you can get the right version of GEOS for each of your projects.
Re:uv downloads overtake Poetry for Wagtail users
by amelius on Monday March 17, 2025 @07:03AM
(#43387170)
Indeed, I'd want something where I have more control over how the binaries are built. I had some segfaults with conda in the past, and couldn't find where the problem was until I rebuilt everything from scratch manually and the problems went away.
Nix/guix sound interesting. But one of my systems is an nVidia Jetson system, where I'm tied to the system's libc version (because of CUDA libraries etc.) and so building things is a bit trickier.
Re:uv downloads overtake Poetry for Wagtail users
by dagw on Monday March 17, 2025 @07:37AM
(#43387399)
with uv (and pip) you can pass the --no-binary flag and it will download the source code and build all you dependencies, rather than downloading prebuilt binaries.
It should also respect any CFLAGS and LDFLAGS you set, but I haven't actually tested that with uv.
Re:uv downloads overtake Poetry for Wagtail users
by amelius on Monday March 17, 2025 @12:08PM
(#43389982)
I just tried --no-binary with the torchvision package (on a Jetson system). It failed. Then I downloaded the source and it compiled without problems.
Re:uv downloads overtake Poetry for Wagtail users
by imtringued on Monday March 17, 2025 @07:04AM
(#43387175)
What exactly prevents you from creating your own packages if you want to use your system package manager?
On Alpine and Arch Linux? Exactly nothing.
On Debian/Ubuntu? maybe the convoluted packaging process, but that's on you for choosing those distributions.
Re:uv downloads overtake Poetry for Wagtail users
by amelius on Monday March 17, 2025 @07:17AM
(#43387258)
On Nvidia/Jetson systems, Ubuntu is dictated by the vendor.
Re:uv downloads overtake Poetry for Wagtail users
by macNchz on Monday March 17, 2025 @08:40AM
(#43387890)
This type of situation is why I use Docker for pretty much all of my projects—single package managers are frequently not enough to bootstrap an entire project, and it’s really nice to have a central record of how everything needed was actually installed. It’s so much easier to deal with getting things running on different machines, or things on a single machine that have conflicting dependencies.
Re:uv downloads overtake Poetry for Wagtail users
by OutOfHere on Monday March 17, 2025 @06:10PM
(#43393240)
Docker is good for deployment, but devcontainer is nice for development. Devcontainer uses Docker under the hood. Both are also critically important for security isolation unless one is explicitly using jails.
Re:uv downloads overtake Poetry for Wagtail users
by dagw on Monday March 17, 2025 @07:05AM
(#43387185)
UV is not (yet) a build system and does not get involved with compiling code. But easily lets you plug in any build system you want. So it will let you keep using whatever system you are currently using for building your C libraries. For example I use scikit-build-core for building all of my libraries C and C++ components with cmake and it works fine with uv.
Re:uv downloads overtake Poetry for Wagtail users
by datadeft on Monday March 17, 2025 @07:11AM
(#43387223)
uv build
Building source distribution...
running egg_info
writing venv.egg-info/PKG-INFO
Successfully built dist/venv-0.1.0.tar.gz
Successfully built dist/venv-0.1.0-py3-none-any.whl
Re:uv downloads overtake Poetry for Wagtail users
by dagw on Monday March 17, 2025 @07:27AM
(#43387327)
I guess it depends on what you mean by a build system. From my understanding uv build basically just bundles up all the source code it finds, and packages it into a .whl with the correct metadata. It cannot actually do any build steps like running commands to compile or transform code or data in any way. For that you need something like setuptools or scikit-build or similar. All of which integrate seamlessly with uv.
Re:uv downloads overtake Poetry for Wagtail users
by datadeft on Monday March 17, 2025 @09:06AM
(#43388160)
I see what you mean. You can use it with mise that has build support.
Re:uv downloads overtake Poetry for Wagtail users
by sirfz on Monday March 17, 2025 @09:29AM
(#43388395)
It actually does exactly what pip does depending on your configured build backend, so if you have your pyproject.toml/setup.py configured to build external modules, `uv build` will run that and build a binary wheel
Re:uv downloads overtake Poetry for Wagtail users
by dagw on Monday March 17, 2025 @09:33AM
(#43388440)
Yes, that's my point. You need to bring your own 'real' build system to make uv doing anything non-trivial. And the fact that this work transparently with uv is a very good thing.
Re:uv downloads overtake Poetry for Wagtail users
by sirfz on Monday March 17, 2025 @09:31AM
(#43388423)
Yes it'll build any dependency that has no binary wheels (or you explicitly pass --no-binary) as long as said package supports it (i.e. via setup.py/pyproject.toml build-backend). Basically, just like pip would
And what if there are no binaries yet for my architecture, will it compile them, including all the dependencies written in C?