I’ve always worked around that by having a requirements.base.txt and a requirements.txt for the locked versions. Obviously pip doesn’t do that for you but it’s not hard to manage yourself.

Having said that, I’m going to give uv a shot because I hear so many good things about it.

I’m grouchy because I finally got religion on poetry a few years ago, but the hype on uv is good enough that I’ll have to give it a shot.

I freaking love Poetry. It was a huge breath of fresh air after years of pip and a short detour with Pipenv. If uv stopped existing I’d go back to Poetry.

But having tasted the sweet nectar of uv goodness, I’m onboard the bandwagon.

With the new major release of Poetry that just came out I also feel like it might be a good time to switch to Uv rather than adapt to this new version: https://python-poetry.org/blog/announcing-poetry-2.0.0/

With pip the best practice is to have a requirements.txt with direct requirements (strictly or loosely pinned), and a separate constraints.txt file [1] with strictly pinned versions of all direct- and sub-dependencies (basically the output of `pip freeze`). The latter works like a lock file.

[1] https://pip.pypa.io/en/stable/user_guide/#constraints-files

For direct requirements you're better off using the `pyproject.toml` for direct dependencies (and you can plausibly use it to pin everything if you're developing an application). It's project metadata that you'll need anyway for building your project, and the "editable wheel" hack allows Pip to use that information to set up an environment for your code (via `pip install -e .`).

This works until you need to upgrade something, pip might upgrade to a broken set of dependencies. Or if you run on a different OS and the dependencies are different there (because of env markers), your requirements file won't capture that. There are a lot of gotchas that pip can't fix.

> pip might upgrade to a broken set of dependencies.

I'm only aware of examples where it's the fault of the packages - i.e. they specify dependency version ranges that don't actually work for them (or stop working for them when a new version of the dependency is released). No tool can do anything about that on the user's end.

> Or if you run on a different OS and the dependencies are different there (because of env markers), your requirements file won't capture that. There are a lot of gotchas that pip can't fix.

The requirements.txt format is literally just command-line arguments to Pip, which means you can in fact specific the env markers you need there. They're part of the https://peps.python.org/pep-0508/ syntax which you can use on the Pip command line. Demo:

  $ pip install 'numpy;python_version<="2.7"'
  Ignoring numpy: markers 'python_version <= "2.7"' don't match your environment

> There are a lot of gotchas that pip can't fix.

There are a lot of serious problems with Pip - I just don't think these are among them.

You can specify markers in the requirements file you write, not in the frozen requirements from 'pip freeze'. Because it's just a list of what's installed on your machine.

Running 'pip freeze' creates a plain text file. You can edit it to contain anything that would have been in "the requirements file you write". "Your requirements file" may or may not capture what it needs to, depending on how you created it. But Pip supports it. (And so does the `pyproject.toml` specification.)

You will need another tool to write a lock file that actually locks dependencies for more environments than your own. I don't know what you're trying to say. Pip does not support writing it.

Sure, I guess if you have one Pip will "support" reading it.

The requirements.txt file is the lockfile. Anyways, this whole obsession with locked deps or "lockfiles" is such an anti-pattern, I have no idea why we went there as an industry. Probably as a result of some of the newer stuff that is classified as "hipster-tech" such as docker and javascript.

Just because you don't understand it, it's ok to call it an "anti-pattern"?

Reproducibility is important in many contexts, especially CI, which is why in Node.js world you literally do "npm ci" that installs exact versions for you.

If you haven't found it necessary, it's because you haven't run into situations where not doing this causes trouble, like a lot of trouble.

Just because someone has a different perspective than you doesn't mean they don't "understand".

Lockfiles are an anti-pattern if you're developing a library rather than an application, because you can't push your transitive requirements onto the users of your library.

If you're developing a library, and you have a requirement for what's normally a transitive dependency, it should be specified as a top-level dependency.

The point is that if I'm writing a library and I specify `requests == 1.2.3`, then what are you going to do in your application if you need both my library and `requests == 1.2.4`?

This is why libraries should not use lockfiles, they should be written to safely use as wide a range of dependencies' versions as possible.

It's the developers of an application who should use a lockfile to lock transitive dependencies.

The lock file is for developers of the library, not consumers. Consumers just use the library’s dependency specification and then resolve their own dependency closure and then generate a lock file for that. If you, as a library developer, want to test against multiple versions of your dependencies, there are other tools for that. It doesn’t make lock files a bad idea in general.

As another library developer, of course I want to test against multiple versions. Or more accurately, I don't want to prevent my users from using different versions prematurely. My default expectation is that my code will work with a wide range of those versions, and if it doesn't I'll know - because I like to pay attention to other libraries' deprecations, just as I'd hope for my users to pay attention to mine.

Lockfiles aren't helpful to me here because the entire point is not to be dependent upon specific versions. I actively want to go through the cycle of updating my development environment on a whim, finding that everything breaks, doing the research etc. - because that's how I find out what my version requirements actually are, so that I can properly record them in my own project metadata. And if it turns out that my requirements are narrow, that's a cue to rethink how I use the dependency, so that I can broaden them.

If I had a working environment and didn't want to risk breaking it right at the moment, I could just not upgrade it.

If my requirements were complex enough to motivate explicitly testing against a matrix of dependency versions, using one of those "other tools", I'd do that instead. But neither way do I see any real gain, as a library developer, from a lock file.

>If I had a working environment and didn't want to risk breaking it right at the moment, I could just not upgrade it.

The point of a lockfile is to only upgrade when you want to upgrade. I hope you understand that.

Why do I need a special file in order to not do something?

That’s not the perspective that OP was sharing, though.

You literally phrased it as "I have no idea why". You can't be upset if someone feels you don't understand.

"I have no idea why" the industry went there. One can understand a technology or a design pattern yet think it's completely idiotic. (low-hanging fruit: JavaScript, containers, etc.)

I’m pretty sure it was a sarcasm.

"pip freeze" generates a lockfile.

No, that generates a list of currently installed packages.

That’s very much not a lock file, even if it is possible to abuse it as such.

A list of currently installed packages in the current environment, with their exact versions. This is only the actually needed packages, with their transitive dependencies, unless you've left something behind from earlier in development. If you're keeping abstract dependencies up to date in `pyproject.toml` (which you need to do anyway to build and release the project), you can straightforwardly re-create the environment from that list and freeze whatever solution you get (after testing).

Doesn’t account for differences in platforms or Python versions, and doesn’t contain resolved dependency hashes.

So it’s a “lockfile” in the strictest, most useless definition: only works on the exact same Python version, on my machine, assuming no dependencies have published new packages.

Look, I'm not trying to sell this as a full solution - I'm just trying to establish that a lot of people really don't need a full solution.

>only works on the exact same Python version

It works on any Python version that all of the dependencies work on. But also it can be worked around with environment markers, if you really can support multiple Python versions but need a different set of dependencies for each.

In practical cases you don't need anything like a full (python-version x dependency) version matrix. For example, many projects want to use `tomllib` from the standard library in Python 3.11, but don't want to drop support for earlier Python because everything else still works fine with the same dependency packages for all currently supported Python versions. So they follow the steps in the tomli README (https://github.com/hukkin/tomli?tab=readme-ov-file#building-...).

>on my machine

(Elsewhere in the thread, people were trying to sell me on lock files for library development, to use specifically on my machine while releasing code that doesn't pin the dependencies.)

If my code works with a given version of a dependency on my machine, with the wheel pre-built for my machine, there is no good reason why my code wouldn't work on your machine with the same version of the dependency, with the analogous wheel - assuming it exists in the first place. It was built from the same codebase. (If not, you're stuck building from source, or may be completely out of luck. A lockfile can't fix that; you can't specify a build artifact that doesn't exist.)

This is also only relevant for projects that include non-Python code that requires a build step, of course.

>assuming no dependencies have published new packages.

PyPI doesn't allow you to replace the package for the same version. That's why there are packages up there with `.post0` etc. suffixes on their version numbers. But yes, there are users who require this sort of thing, which is why PEP 751 is in the works.

So many misunderstandings here :/ I can’t muster the energy to correct them past these two obvious ones

> It works on any Python version that all of the dependencies work on

No, it doesn’t. It’s not a lockfile: it’s a snapshot of the dependencies you have installed.

The dependencies you have installed depend on the Python version and your OS. The obvious case would be requiring a Linux-only dependency on… Linux, or a package only required on Python <=3.10 while you’re on 3.11.

> PyPI doesn't allow you to replace the package for the same version

Yes and no. You can continue to upload new wheels (or a sdist) long after a package version is initially released.

>So many misunderstandings here :/

I've spent most of the last two years making myself an expert on the topic of Python packaging. You can see this through the rest of the thread.

>No, it doesn’t. It’s not a lockfile: it’s a snapshot of the dependencies you have installed.

Yes, it does. It's a snapshot of the dependencies that you have installed. For each of those dependencies, there is some set of Python versions it supports. Collectively, the packages will work on the intersection of those sets of Python versions. (Because, for those Python versions, it will be possible to obtain working copies of each dependency at the specified version number.)

Which is what I said.

> The dependencies you have installed depend on the Python version and your OS. The obvious case would be requiring a Linux-only dependency on… Linux, or a package only required on Python <=3.10 while you’re on 3.11.

A huge amount of packages are pure Python and work on a wide range of Python versions and have no OS dependency. In general, packages may have such restrictions, but do not necessarily. I know this because I've seen my own code working on a wide range of Python versions without making any particular effort to ensure that. It's policy for many popular packages to ensure they support all Python versions currently supported by the core Python dev team.

Looking beyond pure Python - if I depend on `numpy==2.2.1` (the most recent version at time of writing), that supports Python 3.10 through 3.13. As long as my other dependencies (and the code itself) don't impose further restrictions, the package will install on any of those Python versions. If you install my project on a different operating system, you may get a different wheel for version 2.2.1 of NumPy (the one that's appropriate for your system), but the code will still work. Because I tested it with version 2.2.1 of NumPy on my machine, and version 2.2.1 of Numpy on your machine (compiled for your machine) provides the same interface to my Python code, with the same semantics.

I'm not providing you with the wheel, so it doesn't matter that the wheel I install wouldn't work for you. I'm providing you(r copy of Pip) with the package name and version number; Pip takes care of the rest.

>You can continue to upload new wheels (or a sdist) long after a package version is initially released.

Sure, but that doesn't harm compatibility. In fact, I would be doing it specifically to improve compatibility. It wouldn't change what Pip chooses for your system, unless it's a better match for your system than previously available.

Holy hell dude, you don’t need to write a novel for every reply. It’s not a lockfile because it’s a snapshot of what you have installed. End of.

It doesn’t handle environment markers nor is it reproducible. Given any non-trivial set of dependencies and/or more than 1 platform, it will lead to confusing issues.

Those confusing issues are the reason for lock files to exist, and the reason they are not just “the output of pip freeze”.

But you know this, given your two years of extensive expert study. Which I see very little evidence of.

>It’s not a lockfile because it’s a snapshot of what you have installed.

I didn't say it was. I said that it solves the problems that many people mistakenly think they need a lockfile for.

(To be clear: did you notice that I am not the person who originally said "'pip freeze' generates a lockfile."?)

>It doesn’t handle environment markers nor is it reproducible.

You can write environment markers in it (of course you won't get them from `pip freeze`) and Pip will respect them. And there are plenty of cases where no environment markers are applicable anyway.

It's perfectly reproducible insofar as you get the exact specified version of every dependency, including transitive dependencies.

>Given any non-trivial set of dependencies and/or more than 1 platform, it will lead to confusing issues.

Given more than 1 platform, with differences that actually matter (i.e. not pure-Python dependencies), you cannot use a lockfile, unless you specify to build everything from source. Because otherwise a lockfile would specify wheels as exact files with their hashes that were pre-built for one platform and will not work on the others.

Anyway, feel free to show a minimal reproducible example of the confusion you have in mind.

> Given more than 1 platform, with differences that actually matter (i.e. not pure-Python dependencies), you cannot use a lockfile, unless you specify to build everything from source. Because otherwise a lockfile would specify wheels as exact files with their hashes that were pre-built for one platform and will not work on the others.

What is more likely:

1. Using a lockfile means you cannot use wheels and have to build from source

2. You don’t know what you’re talking about

(When deciding, keep in mind that every single lockfile consuming and producing tool works fine with wheels)