I've had a good experience with pip-tools (https://github.com/jazzband/pip-tools/) which takes a requirements.in with loosely-pinned dependencies and writes your requirements.txt with the exact versions including transitive dependencies.

Same here, in my team we had immediate dependencies defined in setup.cfg when PR was merged, a pip-compile was run and generated requirements.txt and store it in central database (in our case it was consul because that was easiest to get without involving ops).

pip-sync was then called to install it in given environment, any promotion from devint -> qa -> staging -> prod, was just copying the requirements.txt from environment earlier and calling pip-sync.

Take my upvote. This has helped us a ton. So nice that it resolves dependencies. Only issue we're running into is that we don't use it to manage our dependencies for our internal packages (only using it at the application level). I've been advocating we change so that we simply read in the generated requirements.txt/requirements-dev.txt in setup.py

Late to the party but `pip-tools` also has a flag for its `pip-compile` flag: `--generate-hashes`. It generates SHA256 hashes that `pip install` checks.

If you have "foo==0.1" installed, and foo has the dep "bar~=0.2" (current vers of bar 0.2.1).

Then bar releases version 0.2.2

So your deps want bar 0.2.1, but foo now wants bar 0.2.2

This breaks your pip install.

EDIT: there are a few other gotchas (please respond to this post if you know of any more)

e.g. from https://medium.com/knerd/the-nine-circles-of-python-dependen...

"If two of your dependencies are demanding overlapping versions of a library, pip will not necessarily install a version of this library that satisfies both requirements" e.g. https://github.com/pypa/pip/issues/2775

Another more obscure one:

https://github.com/pandas-dev/pandas/issues/27206

All of a sudden, a numpy release pulls in a new version for a pandas build (that incidentally breaks for py2)

This without involving a "~=", but rather because pandas needs to build from source, and chooses the latest numpy build to do so.

This is what I've always done. Develop using a few dependencies, freeze, continue development with reproducible builds. It has always included the sub-dependencies in the list so, as far as I can tell, this works great for that case...

It's not functional for library dependencies, where you still need to manage the setup.py instead. Only useful for end-user application dependencies.

I think he is referring to indirect dependencies

Yes, those are caught when using pip freeze.

> Yes, those are caught when using pip freeze.

No they are not.

Pip freeze does not resolve transitive dependencies, nor does pip know what to do if your transitive dependencies conflict with each another.

> Pip freeze does not resolve transitive dependencies

I don't think this is correct:

    $ python3 -m venv /tmp/v
    $ /tmp/v/bin/pip install flask
    [...]
    Collecting MarkupSafe>=0.23 (from Jinja2>=2.10.1->flask)
    [...]
    $ /tmp/v/bin/pip freeze | grep MarkupSafe
    MarkupSafe==1.1.1

> nor does pip know what to do if your transitive dependencies conflict with each another

This is true, but because Python exposes all libraries in a single namespace at runtime, there isn't actually anything reasonable to do if they genuinely conflict. You can't have both, say, MarkupSafe 1.1.1 and MarkupSafe 1.1.0 in PYTHONPATH and expect them to be both accessible. There's no way in an import statement to say which one you want.

However, it's notable that pip runs into trouble in cases where transitive dependencies don't genuinely conflict, too. See https://github.com/pypa/pip/issues/988 - this is a bug / acknowledged deficiency, and there is work in progress towards fixing it.

> There's no way in an import statement to say which one you want.

This would be fixable with a sys path hook, were pip so inclined

It would change the semantics of the language. You could also write a sys.path hook to interpret the remainder of the file as Ruby and not Python, were pip so inclined....

(Also it's not clear what those changed semantics would be.)

Import system is pluggable, so the semantics are there to be customized. Sure, it could be abused (as many things in Python), but an import hook that checks for a vendored dependency with specific version, seems like a reasonable way to resolve the problem above.

> remainder of the file as Ruby and not Python

That's a little excessive

But it changes the semantics of the rest of the language, e.g., if two modules interoperate by passing a type of a third module between themselves, and now there are two copies of that third module, they can't communicate any more.

Getting this right and reliable would be a) a considerable language design project in its own right and b) confusing to users of Python as it is documented, and in particular to people testing their modules locally without pip. It wouldn't be as drastically different a language as Ruby, but it would certainly be a different language.

>Pip freeze does not resolve transitive dependencies

How? Doesn't pip freeze literally list all packages that's installed in the current environment besides basic toolings such as setuptools (and you could even instruct it to list those as well)?

I'm not sure about conflict resolution when but I run a pip freeze it adds TONS of dependencies outside of the 2-3 I had in my app because those were the dependencies of my dependencies.

I think that is what you want. Having all your dependencies, including their dependencies, explicitly specified (including name and version) is what gives you reproducible builds.

Ruby does the same thing with Gemfile.lock. npm does the same thing with package-lock.json.

They are, but pip doesn't know if they are transitive dependencies or not or if they conflicted at install time. Pip-tools helps with this.

>>> Why isn't there any builtin way to automatically define a lock file

pip isn't actually part of Python proper.

Nowadays, a regular installation of CPython lets you run "python -m pip". It's quite part of it.

You can do that with any library. You can issue Django commands by running `python -m django`; that doesn't change the fact that Django is a completely separate project from Python.

It isn't part of the Python executable, but it is part of the standard distribution.

It catches way too much. IPython, black and the testing libraries are _not_ a part of my actual dependencies and shouldn't be installed in production. A good UI for a dependency manager at the very least distinguishes between dev and production context, and ideally lets me define custom contexts.

The only problem with that is it's hard to keep the dependencies up to date. Pip-tools solves this problem.

Why do you want to update your dependencies if they work? Isn't the whole point of dependency management to avoid using different versions of dependencies than the ones they have been tested on?

Security fixes, performance enhancements, new features. There are many reasons. But the point is you update in a controlled manner. You don't just push the latest version of everything out on to prod, but you also don't keep pushing the same version that worked a year ago.

You want to easily be able to update dependencies (and retest with them, of course) to ensure you’re getting security updates, for one.