Only pdm and poetry generate cross-platform lock files by default as far as I know, but there are a lot of people trying to solve this problem right now.
It's not an easy problem to solve. Python's package management predates package managers from most other programming languages and Python itself predates Linux. There is a lot of baggage so change is very slow.
Standards ensure that going forward language semantics and syntax don't change. Having minimal dependencies ensures program longevity. Package manager cannot solve these problems, no matter how good it is at its job.
Python doesn't have a standard, it's heavily reliant on dependencies which are very plentiful and similarly unregulated. A program written in C that uses only functionality described in some POSIX standard will endure decades unmodified. Even Python helloworld program went stale sometime ago, even though it's just one line.
Or the "says it will work with > X," but doesn't.
This is, of course, an infrastructure/maintenance issue as much as a package manager design issue. But in Nix's case, the public 'binary cache' (for Nixpkgs/NixOS) of build outputs includes not only final build outputs but also the source tarballs that go into them. As Nix disallows network access at build time, all dependencies are represented this way, including jar files or source tarballs, or whatever— Nix itself must be the one to fetch your dependencies. Consequently, everything you fetch from the Internet for your build is a kind of intermediary Nix build that can be cached using the usual Nix tools. The Nix community's public cache has a policy of retaining copies of upstream sources forever (there is recently talk of limiting storage of the final built packages to a retention period of only 2 years, but sources will continue to be retained indefinitely. So far the cache reaches back to its inception around a decade ago.)
Taken together, these things mean that when a project disappears entirely from GitHub or Maven Central or whatever, people building against old versions of it with Nix/Nixpkgs don't even notice. Nix just fetches those upstream sources from the public cache without even reaching out to that central repository from which those sources have been removed.
For private use cases where your project and its dependencies won't be mirrored to the public cache of Nixpkgs builds, you can achieve the same effect by running your own cache or paying a hosted service to do that.
For builds outside the Nix universe, you can make special arrangements for each type of package your various builds fetch, and mirroring those repos. Then configure your builds to pull from your mirrors instead of the main/public ones.
Nix does, if you want to actually invest the effort into the "without asterisks" part.
python -m venv myenv
. myenv/bin/activate
<install reqs>
<your program>
But not the same Python runtime, which does not fulfil the request of guaranteeing it works the same later without asterisks.
Just don't uninstall your python binaries and you're fine.
If you want to package everything up into am image or a zip, you can do that too if you want, but in my 10+ year career I've not had much of an issue just using boring venvs.
It seems that there is a way to do this in pip but I don’t think it is widely used: https://stackoverflow.com/questions/19559247/requirements-tx...
It does not.
> There are symlinks in the environments bin directory to the specific runtime.
Precisely. They symlink to a path. Which means that if you have a certain version of Python at one location (let’s say /usr/bin/python3) and later update that (let’s say by upgrading macOS and the Xcode developer tools), the same virtual environment will point to a different version of Python.
> Just don't uninstall your python binaries and you're fine.
That’s not a practical solution. Sometimes you don’t have a choice, as demonstrated above. By that logic one could say “don’t change anything about your system and you don’t even need virtual environments”. Which is somewhat true, but also profoundly unhelpful.
The point of the question was to reproduce the same thing without asterisks and you’ve introduced a major one and called it a day.
It's python3.11, not python3, and the python3 "executable" is, itself a symlink to the particular binary (in this example python3.11). Upgrading just changes the symlink, which wouldn't affect the venv, which isn't using python3, it's using python3.11.
I didn't introduce anything, I explained how the links are to the versioned binaries, which isn't what you're stating is happening.
Edit to add: as someone else also points out, you don't even have to use the symlinked versions, you can use --copies
Not for the example I gave. If you’re seeing Python 3.11, you’re definitely not using the /usr/bin/python3 on macOS with is made available by the Xcode CLI tools. That’s at 3.9.6 even on Sonoma.
> as someone else also points out, you don't even have to use the symlinked versions, you can use --copies
Which, again, doesn’t work for the stated case.
No, it is you who are failing to understand. I’m describing to you a real scenario, but it’s not one that bothers me. I don’t need you to come up with a solution and didn’t ask you for it. You need to understand not everyone has the same requirements and tradeoffs you do.
> Error: This build of python cannot create venvs without using symlinks
— Then don’t raise your arm.
Like I said in the original comment you replied to and are now ignoring:
> That’s not a practical solution. Sometimes you don’t have a choice, as demonstrated above. By that logic one could say “don’t change anything about your system and you don’t even need virtual environments”. Which is somewhat true, but also profoundly unhelpful.
- Doctor, it hurts when I hit my head on the wall
- Then don't hit your head on the wall
You have solutions available, use them
— Hey, so you know how we have this requirement, which came about after years of dealing with and understanding a problem and the available solutions?
— Yes, what about it?
— Well, a random commenter on Hacker News, which has zero context of the problem, has suggested we use a method we already found inadequate for our specific use case.
— Oh wow, in that case let’s replace the whole system right now.
I'm not ignoring you; I just don't think your use case of "I want to capture the Python from Xcode in my dev env so it's resilient to changes and upgrades" is something anyone wants (or should want) to do. Do you really want to ship on that version? Are you asking your users to install Xcode?
> That’s not a practical solution. Sometimes you don’t have a choice, as demonstrated above. By that logic one could say “don’t change anything about your system and you don’t even need virtual environments”. Which is somewhat true, but also profoundly unhelpful.
In other words: when are you forced to use Xcode's Python for development, and why would that be a good idea? I'm earnestly asking; there may be reasons; I'm just not aware of any.
Then you are wrong. Simple as that. I’m describing a very real scenario.
> Do you really want to ship on that version?
Holy moly, is it really that hard to understand the difference between wanting and having to? For the use case, and older version in a consistent place which is easy to install is the best solution.
> Are you asking your users to install Xcode?
Triggering an Xcode CLI tools installation is simple and done graphically. And it’s one step removed from installing Homebrew or pyenv, which both need them (even for the scripted installation, pyenv requires git).
> In other words: when are you forced to use Xcode's Python for development, and why would that be a good idea?
See, for a moment there you understood it’s about users, not just your own dev environment, but then went back. Unfortunately, after this conversation with you two I no longer have the energy to go through it in detail in an uphill explanation. Another time, maybe.
> Then you are wrong. Simple as that. I’m describing a very real scenario.
>> Do you really want to ship on that version?
> Holy moly, is it really that hard to understand the difference between wanting and having to? For the use case, and older version in a consistent place which is easy to install is the best solution.
Tone is hard on the internet, but I'm honestly trying to understand your use case. The way I understand it now is "I want to be able to develop Python programs locally using only Xcode's Python." I'm still not sure why you want to (generally when people ship Python programs they bundle a runtime) but let's set that to the side. For this use case, I don't understand why symlinks don't work for you. Xcode installs to versioned folders so you can have multiple Xcode installs side by side, that way new versions won't overwrite things. I'm obviously not an expert here though; am I missing something?
I never said that. My assertion was that venv does not ensure the same Python runtime.¹ That’s it. I don’t have a problem with that. But I do know of one situation where it could make a difference and “do it another way” is not a reasonable answer.
If we ever meet in person I’ll gladly explain it in detail.
You're supposed to install a specific version of python in a specific place, with a specific name. Say, /usr/local/python-3.10.6
Use pyenv to use that python. Control that by creating a `.python-version` file that says 3.10.6
You now have a project that uses 3.10.6. Unless, of course, somebody installs a different version in that path - at which point you've got bigger issues
Using pyenv to use `/usr/bin/python3` and hoping for the best misses the point
Because that’s not what the conversation is about. It’s about virtual environments.
And that's the whole problem. "The same Python environment" doesn't mean much because it underspecifies the full runtime dependency chain.
No other solution besides fixed OS (containers, nix) could solve this problem.
You don't have to use NixOS to use Nix. You can install it on any Linux distro, or MacOS, and use it as your build system.
Forget to install virtualenv? Have fun getting rid of packages in your global system.
Sub-dependencies? You'll have to think of a strategy for it. Most people don't and the reproducibility of their project suffers.
Developer vs production dependencies ? You'll have to think of a strategy for it.
And I think I'm forgetting one, but that's already enough.
For me, the combination of asdf and Poetry has worked quite well recently: I use asdf to pin the Python & Poetry version and then use Poetry to pin everything else.
https://github.com/asdf-community/asdf-python?tab=readme-ov-...
We still need a generated lock file with every top level dependency and sub-dependencies locked down to their most precise version commit to version control so that when you build your image today or in 6 months you end up with the same result.
Using pip to freeze your dependencies and writing a tiny shell script to generate a lock file at build time is better than nothing to solve this problem with nothing more than pip. It's what I do in https://github.com/nickjj/docker-flask-example and https://github.com/nickjj/docker-django-example. It's not perfect but it solves 80% with minimal complexity.
Of course this doesn't come free: packaging with Nix may involve nontrivial effort for some projects.
nixpkgs maintainers are frequently the first to notify a project author of incompatibility with new versions of another package.
In Nix lingo, a package successfully building or running against an implicit or unmanaged dependency is called 'impurity'. To help keep builds 'pure', Nix builds everything in a sandbox and does various other tricks to ensure that a package being built can't/doesn't find any dependencies at build time that you don't explicitly tell Nix to include in the build environment.
(This is also increasingly how Linux distros build their packages, to solve the same problem.)
For the most part, if building (and running the test suite during the build) a Python package with Nix succeeds, you can be confident that you've got all the dependencies sorted out— even the ones upstream forgot to tell you about.
> Do you have to find them out and fix them manually or can you generate a lock file and hope for the best?
At install time, you can be confident that Nix will bring along all of the system-level dependencies your Python package needs. You don't have to find and fill any gaps at install time 10 years from now or whatever.
When you're writing your Nix package for the first time, you'll be doing a mix of generating the Nix code that defines your package from some upstream, Python-specific lockfile and making manual corrections when the build fails. Nix doesn't have any magic for figuring out dependencies that are left out of poetry.lock or whatever, or for disambiguating guaranteed-compatible exact versions from requirements.txt.
Using python and python libraries only from your package manager (like APT) for a specific OS version.
Micromamba as well, lightweight version of conda/miniconda.
I think rye hits most of the points pretty well, it ensures both python and package versions.
Edit: Wow y'all are some sour people for voting down this question. I truly wonder how often people run into this problem compared to just complaining about it.
In my edperience I had many many problems with OS packages vs pip installed ones. There were really strange dependency issues.
In some cases I encountered dependency hell.
Even if somebody said to me that the core means also using virtualized en I would disagree as places over Internet not always explicitly guide you in that route.
NB. The latest incident was Friday when I've discovered that some CI pipeline ran `setup.py install` that down the lane invoked easy_install, which doesn't have a policy of ignoring bizarre versions s.a. X.Y.Zrc1 or X.Y.Zb2 etc. It ran aground when it was trying to install scikit-learn which wanted NumPy >=X.Y.Z, but it already installed X.Y.Zb1, and it didn't realize that this version should be OK (also, it shouldn't have installed non-release versions anyways).
I run into some kind of packaging or dependency problem for basically every non-trivial project. Hence the many "solutions", but somehow most end up worse than the problems they attempt to solve
It’s no panacea, but feels more stable and usable (especially from onboarding new team members PoV) than other tooling I’ve tried
The generated virtual env name is a little wonky. I'm not sure exactly what the scheme is, but it's basically `$(package)-$(some sort of hash?)-$(pyversion)`. I can't speak for all tools, but at least VS Code detects the poetry env and suggests it (and indicates it as a Poetry env) when you go to configure the project interpreter.
There aren't such tools. Python not being a standard and heavily reliant on the OS that runs it and on third-party components that are also not standard leaves you with no choice by to "be at the wheel" all the time. Virtually anything written in Python will go stale in a mater of few years. In other words, you need to constantly update and test as the environment changes just to stand still.
The other one is Docker, which some people hate. I don't mind it, in some cases I prefer it.
requirements.txt is a text file, not a separate tool...
if you want your program to work exactly as intended in the future, there are tools like py2exe and py2app for that.
On my personal machine I don't bother with venvs, so also haven't thought about it in years. But was trying to figure out what the "workon" command did for GP.
this way I have all my envs in ~/virtual and all my projects in ~/projects and just `workon xyz` when I want to be in a certain venv for a given project (which doesn't always map one-to-one)
Edit: Oh I see you didn't. I can't read.
Which ones did I miss? Which of them actually ensure your program always works the same as when you first wrote it, without asterisks?