Your lock file goes into version control. This ensures all your developers and environments are using the exact same version of every library. You generate a new lock file when upgrading/adding libraries.

Don't be so scared. Ruby developers have been doing this for over 7 years now. Trust us, this is much better. Your Python friends at Pypa are copying Ruby because dependencies are less painful there than just about anywhere. They are still painful, but this will help.

I think the issue here is that it adds so much complexity. There's a project called pip-tools (https://github.com/nvie/pip-tools) that does a similar thing (generates a requirements.txt, test-requirements.txt etc) from a requirements.in files, so it serves the same purpose as a .lock but is backward compatible, much simpler, not subject to abuse by having it be actual code, and is machine parsable by any language.

Having a more complex requirements system that what we have with requirements.txt is good, but at what cost? Is Python doing this just because other languages to it this way? I think it is actually

The purpose of a lock file is to separate the concepts of "acceptable versions" from "the list of last exact versions I used and were working". These are very different things and requirements.txt doesn't do both without you getting really anal with your version requirements (and losing the benefits of less strict version requirements).

With pip-tools, requirements.txt is your lock file, and everything in it is pinned. It's built from an acceptable-versions input usually called "requirements.in".

Interesting. I guess Pypa feels Python also needs a flexible DSL and dependency groups in addition to the pip-tools lock file solution.

Probably, yeah. Though unless it's Real Python™ code, and can do lots of shenanigans, I don't see how it's more flexible. And all those shenanigans mean :'( in the same ways as setup.py. And if it's not Real Python™ and just a python-like declarative syntax, then why not just add the features to requirements.txt and the command-line, and maintain that parity?

Dependency groups don't really mean an advantage to me. E.g. with pip-tools, on the code I work on, we've got 3 requirements*.txt files. One (requirements.txt) for prod, and ones for test/dev/any additional scopes you may want. Then you `-r requirements.txt` in requirements-dev.in (and -test), and you're guaranteed to maintain the same versions as production when resolving requirements-dev.txt, or have conflicts if something dev adds prevents that from working. In CI / build / etc you just install the single file that's relevant to you.

Bundling groups into the file would be nice for not being able to make mistakes (our approach above requires you to compile things in order, for example), but that would have to weigh pretty heavily against breaking backwards compatibility and a very-simple DSL that already exists.

I'm not sure age is a good indication of goodness in this regard. Python has been doing setup.py for much longer than 7 years. It's a similar format, and ripe for abuse because you have full access to a general purpose programming language in it. I've seen setup.pys call out to Inkscape. I don't want a general purpose programming language in my requirements.txt (or equivalent). I want a dumb declarative data file.

Also, I'm not sure how a lock file in source control will help me when someone who is not checking out source wants to pip install my project using said lock file.

Age is a good indication if there were serious issues with this approach we would have found them.

There seem to be a lot of strange fears from python developers about this approach and my comment on age was an attempt to assuage those fears. Since ruby developers are quite happy with bundler (at least in comparison to other communities) and have been for some time it's a reasonable point and not the only one I made.

I'm not sure how pip plans to use the lock file for projects distributed via pip. They may very well have a solution planned for that.

However, for projects that are distributed by source (which are many) the lock file ensures deterministic builds, and requirements.txt generally doesn't without being pedantic with your versions.

> There seem to be a lot of strange fears from python developers

There you go again.

This isn't fear. It's asking the question of: What. Is. The Damn. Point. Of. This?

I have yet to see a simple, cogent explanation of why this is better than a list of requirements specifying;

pyside==x.y.z

pillow==x.y.z

As opposed to this proposed Thing.

You know what this reminds of?

It reminds me of replacing super simple .INI text files with configuration files stored as XML - change for the sake of it, because Change!

Now, I'd love to see a very simple explanation and justification on why we should all move to this new Thing you're making.

Something like "This new Thing is better because..." , followed by practical /examples/ , because right now, all I see is More Complicated Stuff For The Sake Of It.

There were clear fears/risks/concerns stated by witten directly above. I didn't invent the idea of fears being stated out of whole cloth.

> subject to all the abuse you can introduce

> might cause [...] problems

I've said a few things about the potential benefits, though I am not involved in the project. The actual authors offer the best list of benefits: https://github.com/pypa/pipfile#the-concept

They also point out this will eventually be built into pip, so it will have the benefits of all the existing workarounds like pip-tools without any setup. It does seem like a lot more than change for the sake of change, and along with it since there is so much change why not take the opportunity to adopt a flexible DSL built for future extension so there is less real change in the future.

One thing I learned yesterday was that requirements.txt supports conditionals like

    SomeProject==5.4; python_version < '2.7'
    SomeProject; sys.platform == 'win32'

In my view, that's already out of hand, and isn't even really parsable without some bespoke library.

I'm a big fan of this idea of having a requirements.lock. I can call pip to do the parsing, then I can parse the lock which is just json.

The fact that you need to parse the requirements.txt feels like a design issue with pip to me. (Of course, I'm coming from Rubyland.) As for setup.py, setup.py is broken because it conflates build scripting with package metadata and dependencies. Now that tox exists things are better (but the fact that tox is not a general-purpose programming language is IMO unfortunate coming from Rake/Ruby/Bundler/Gemfiles.)

> Don't be so scared.

Was that really necessary? You just moved the tone of your reply to a less pleasant manner.

The poster isn't scared. They, like me, are failing to see the necessity of this compared to the more simple, straightforward, already-working, KISS-principle-following requirements.txt, which was my immediate reaction to reading the description of this project on the github link.

It was an attempt to directly address the FUD about running python code during dependency resolution. You're doing this to run more python code from the project's author so this seemed odd and an irrational fear.

Maybe my tone came across condescending. That wasn't the intent. It was meant in jest in hopes the reader would second guess the fear.

KISS is great, but this project is here because it's not already working for everyone. There is benefit to a flexible dependency system that keeps the concepts of "acceptable versions" separate from "actual versions last used" while also adopting an extendible DSL ready for edge cases that haven't been thought of.

To be clear, I totally get the benefits of formal support for an "actual versions last used". That is a topic near and dear to my heart, having been burnt by floating versions more than once. It's just that I don't want to swallow a too-flexible DSL in order to get that benefit, because I've also been burnt multiple times by using a DSL when a declarative data format will suffice.

And you don't need both. You can pip freeze a frozen_requirements.txt. You can ship around a whole virtual environment. And I can conceive of more requirements.txt 2.0 solutions that don't require over-wrought DSLs.

You guys were really burned by this setup.py stuff, huh? I empathize.

I also find it a little odd since a destructive project author can still find ways to mess things up without any code executing during setup, right?

Since this is for project level dependencies it doesn't seem like the potential for abuse is too high since you'll likely have a single Pipfile per project and the project will control it.

I don't know that anything will prevent destructive developers from being destructive. I would hate to throw out all DSLs because of a few bad actors.

For what it's worth in Ruby even library dependencies are written using a ruby DSL that runs unprotected during install, as well as some post installation hook facilities. There have been minor abuses like annoyingly long post-install messages and other crap that has occurred, but through simple community pressure the offending libraries are eventually pushed back in line and it's not an issue Ruby developers regularly encounter. Communities are different but Python seems like a community that appreciates best practices and is maybe better at enforcing them than Ruby is, so I'm not sure preventing one minor way a terrible developer can bite you is worth hamstringing your build system.

Yes, an over-wrought DSL is not a prerequisite. You can do deterministic builds and even grouped dependencies without it. But complex platform specific dependencies, git based dependencies, and dependency edge cases we haven't yet envisioned are more easily captured in an extendible native DSL than a more static data format. The DSL also has the advantage of being "just Python" so it will be very easy to remember for Python developers.

Maybe the benefits aren't worth the trade-offs but don't be so quick to count potential or perceived trade-offs as actual ones until you try out the new build system and see how it impacts your workflow. I know you got burned but this isn't setup.py and potential for abuse isn't the same as abuse... yet.

It's a matter of opaqueness, not of abuse: the install system will be unable to reason about what an arbitrary script does, severely limiting functionality. For example, pip will be unable to ensure dependencies remain constant (after installing something else, after DST begins or ends, between a dry run and an actual install, after unrelated software alters environment variables, etc.). This is much worse than a 95% correct dependencies manifest that can be easily hacked to 100% correct.

pip will be able to do everything you suggest by using the lockfile.

The whole point is that if you want things to be reproducible you don't even need to look in the Pipfile. requirements.txt already supports conditionals, so the Pipfile.lock is strictly simpler to reason about in that sense.

Can you give some concrete examples of the issues with using a setup.py/DSL? I've only been using python intermittently for a few years and I'm probably not exposed to these things. Am I setting myself up for a fall by using setup.py?

Here's a setup.py that includes an interactive prompt, and thus breaks all ability to do automated installs! https://github.com/pypa/pip/issues/2732

Here's a setup.py that cannot even be imported unless another package (numpy) is already installed: https://github.com/scipy/scipy/blob/master/setup.py

I've seen setup.pys that read random files from the filesystem. I've seen setup.pys that shell out to random system commands that may or may not be installed. It's just a horrible format for what it's trying to accomplish.. because it's not a format! It's a programming language.

Not as such, but I've encountered setups that shell out to Inkscape, ImageMagick and other tools, and don't gracefully handle failure, sometimes to the point where the only error is "Failed to install egg".

> Don't be so scared. Ruby developers have been doing this for over 7 years now. Trust us, this is much better. Your Python friends at Pypa are copying Ruby because dependencies are less painful there than just about anywhere. They are still painful, but this will help.

I think the condescending tone is quite unnecessary ("Trust us, this is much better. Your Python friends at Pypa")..

In my experience Ruby is a a far worse culprit than Python for breaking things. First of all `rvm` and `rbenv` are both bloated and rather unreasonable invasions of the shell (hi-jacking `cd` seriously?) Secondly, the fact that Ruby modules are globally shared -- can cause endless breakage from a few misbehaving packages, especially with ones that like to monkey-patch std-lib classes (refinements should improve the the later but it will take a while for the ecosystem to catch-up).

I agree. Cargo is the new gold standard at least in my regard, I'd personally be happy if this was a reimplementation in Python for Python.

While I agree with the issues that may stem from using code to express dependencies, there are upsides. I think the main deficiency in Python is the need for virtual environments per project (and the fact this requires a separate tool).

If this Pipfile.lock approach allows me to always have the correct package versions without virtualenv, it's a huge step forward.

Since we can reference other requirements with `-r req-dev.txt` inside requirements.txt . Current requirements.txt isn't really machine-readable without firing up a full Python interpreter.