E.g. that practice seems to be quite prevalent in a few high-profile pytorch packages, which caused our team a lot of pain when trying to enable reproducible cross-platform builds 1-2 years ago
How do you pronounce "uv"?
Now its:
* Install Rye,
* Pull from Github,
* Type rye sync
Since Rye already uses uv behind the scenes there won’t be a lot of incompatibilities to deal with so the migration should be trivial.
Question: are there any plans to add sandboxing support to Rye or UV? I realize it is a big ask and that it might not be possible on all platforms.
My dream scenario is that I can define a section in pyproject.toml that goes like
[sandbox]
allowed_urls = [...]
allowed_dirs = [".",...]
https://github.com/cloudflare/sandbox
FWICT, it's not possible to achieve something like this reliably on Windows natively, but maybe in WSL it could work. On Mac, there is the similar libsecinit aka App Sandbox which also can be spawned from a userspace process, I think?
Of course in every case the program spawning the sandbox has to be outside the sandbox. But having to trust Rye or UV is much better than having to trust thousands of "RandomDevsNichePythonPackage".
This is one concern that would prevent the team I'm on from moving. We do ML work, so I'll use a pytorch-based project for my example. The desired state is that all dependencies are in pyproject.toml, e.g., from which you can generate a set of lockfiles from an AArch64 Mac or an AMD64 Windows workstation for the following platform configurations:
1. Mac with default (MPS) pytorch
2. Windows with CUDA pytorch
3. AArch64 Linux with CPU pytorch
4. AMD64 Linux with CPU pytorch
5. AMD64 Linux with CUDA pytorch
P.S. Big thanks for making Ruff, we're huge fans of linting and formatting both.
For example, it used to download wheels for all matching distributions (say pytorch for a few platforms) to get metadata. That would be like a 10 GB download for 4 wheels of pytorch, to generate the lockfile.
Another thing it used to do is ignore local cache, so for some locking operations would download the same wheel multiple times.
In general, it was not entertaining, from an ML practitioner's perspective, to infer an attitude from the Poetry maintainers that the mountain shall come to Muhammad, so to say. Their abridged position was that pytorch is not compliant with standards, when Poetry itself, e.g., is not PEP 440-compliant, in a manner pertinent to the issue at hand. [2]
I realise that my grumbling about pytorch has nothing to do with your perfectly adequate comment, that said. Please interpret it as nothing more than an expanded rationale fore being more interested in learning about Rye/uv than in revisiting Poetry, at least for the projects I've worked on.
You need to go straight into their websites and manually figure out what goes with what.
Quite honestly, this needs to be done in Docker with fancy conditional logic, and not via the python package manager.
Installing it via the PyPI gets you the CPU version on Windows/macOS, and CUDA 12.1 on Linux. You can specify CUDA version explicitly, which you need to get CUDA on Windows, or CUDA 11.8 on Linux, by referencing the corresponding PyTorch package index, e.g., https://download.pytorch.org/whl/cu118 - cu118 means CUDA 11.8. Doing that explicitly is the normal practice, since in ML land you'll know your hardware well - no one is asking for tooling that automatically resolves CUDA/cuDNN/etcetra compatibility.
Hence, it hopefully follows that installing this with pip is trivial, you just need to maintain multiple requirements.txt files so that you specify the correct index URL where it's necessary. What I specifically want, then, is for someone to give me a UX where I don't have to maintain these requirements.txt files. I should be able to just specify arbitrary build environments* in pyproject.toml, and override the global default package index for specific packages in [platform-]specific build environments.
*Really, we're probably talking about combining blocks of optional dependencies into an ersatz build environment, but that's what they ultimately are for.
If I can then "compile" these build environments into properly reproducible lock files, and then use the dependency manager to export these lock files into conventional requirements.txt files with exact pins, then we've arrived at what I would call a good UX for specifying dependencies in Python.
To the best of my knowledge, for 99.9% of packages this doesn't require Docker. Different parts of this process work well across the ecosystem, just not as a matter of full package. To name a few
- Poetry has many building blocks, but just has had a questionable implementation as per parent you're replying to, and also random other problems like not working at all with private indexes back when I tried, and so on;
- pip-tools does the entire pipeline well, but it quite unfortunately doesn't support cross-platform targeting;
So, quite honestly, until someone can do this correctly and in one place - and, ideally, with endorsements from PSF and PyPA - I'll just consider Python dependency management tooling to be inadequate for my use cases.
It's a hacky workaround, but it seems to work so far. It would be much nicer to see this solved in a better way, though!
Setting up a new project is: rye init && rye sync Adding a dep is: rye add flask && rye sync You can pin your python version in the pyproject.toml
Migrating from an established project is a little hard than it should be. Importing the requirements.txt into the pyproject.toml is not a good idea as uv gets itself in a twist with all the low-level dependencies that exist in the requirements.txt. I've never tried it with a poetry made pyproject.toml, report back if you try it.
On the whole its a good experience, fast and straight forward.
Can you explain? I wonder if you mean "requirements.txt generated via pip freeze" rather than "a human curated requirements.txt"
If you just keep the requirements.txt "high level" then you should be okay. Just my experience with uv hanging for ages with a pip freeze made requirements file.
https://pypi.org/project/sbom4python/ https://github.com/CycloneDX/cyclonedx-python
Your recently added ability to unpin dependencies so devs are more encouraged to stay compatible as they dev, then generate a correct explicit requirements.txt for reproducibility, makes both vuln management and the SBOM step a far easier thing than poetry etc.
Thank you!
For similar reasons, we use https://hatch.pypa.io/latest/why/ and appreciate that it plays nicely with `uv`.
Don't get me wrong, both are hilariously bad, but I like to see more chaos in Python infrastructure. You pushing against PyPA (rightfully) delegitimizes them.
It doesn't look like this is currently documented but I found some hints in the release notes.
E.g., we recently added support for "universal" resolution in uv, so you can generate a locked requirements.txt file with a single resolution that works on all platforms and operating systems (as opposed to _just_ the system you're running on). And Rye supports it too in the latest release.
[1] https://github.com/astral-sh/uv
---
I work on Rye and uv, if you have any questions :)