They don't do the same thing. pipenv allows you to create, manage virtual environments(using venv) apart from managing requirements file(Pipfile) and an npm like locking mechanism, dependency graphs, dev dependencies and more.

I prefer poetry though, since the consensus seems to be coming together on pyproject.toml rather than individual files like Pipfile. A lot of tools have already started supporting the toml file for their config, or have PRs pending.

I understand the value of a locking mechanism in the JS ecosystem because 1/ many packages depend on an intricate web of other packages that overlap, 2/ many packages use semver ranges, and 3/ you don't want two versions of the same package running on a user's browser, due to conflicts and increased size.

I can't think of many Python packages that have the same issues, and Python code isn't sent to and running on a user's browser.

Am I wrong or is there a reason that a locking mechanism (other than git) is helpful in Python?

I think this has less to do with "you don't want two versions of the same package running on a user's browser" and more to do with "when I clone a project and run npm/pip install I want it to be in a known state".

I don't use Python/pip much but as for npm: the problem is when your dependencies, direct or indirect (dependencies of dependencies), aren't "exact". You have something like "~1.2.3" or "^1.2.3". If every developer followed Semvar perfectly, never shipped regressions or new bugs when fixing a bug, and was always able to identify every breaking change then life would be perfect.

That is, however, not the world we live in. So a "lock" file respects your "fuzzy" versions ^/~ when you first run the npm install and then subsequent runs will install using the exact versions you downloaded the first time. This helps solve the "works me me"/"work on my machine" problems. The idea being if you can run it locally then the build server and production can also build/run your code.

All of those apply to Python as well except the multiple packages concern has nothing to do with a browser and everything to do with the fact that a given Python process can only load one version of a library at a time (and probably for good reason).

I mean, Rust uses the same locking mechanism to great success. I've never seen a breaking change from upgrading a Rust dependency that preserves semver (which is 99% of them)

> pipenv allows you to create, manage virtual environments(using venv) apart from managing requirements file(Pipfile) and an npm like locking mechanism, dependency graphs, dev dependencies and more.

Why on earth would anyone need all this to manage packages for their projects?

Are there any other programming languages whose package-management comes close to this level of intricacy and complexity?

“This project needs package X”

I mean how hard can that be to get right in a self-contained environment?

This whole story is just madness coupled deep denial and Stockholm-syndrome.

> Are there any other programming languages whose package-management comes close to this level of intricacy and complexity?

I think many do. Ruby, Node, Erlang/Elixir, Java, Go, Rust, dotnet, C... I’m having trouble thinking of a modern language that doesn’t have such package management mechanisms.

For many people doing anything more than writing one-off scripts, and especially for anyone who collaborates with others or shares their code, package management is so much more than just “this project needs package X”.

But all your examples are simple and reliable tools with a minimum of intricacy.

My criticism isn’t about having a package-management story.

It’s about having a terrible and complex one.

Your original comment was responding to “create, manage virtual environments(using venv) apart from managing requirements file(Pipfile) and an npm like locking mechanism, dependency graphs, dev dependencies and more.” and saying that this is overly complicated. Yet every single one of the listed languages has package management tools that do all of these things.

If you think any of the listed examples have “simple” package management tools, I question how deeply you have used any of them. NPM has ~60 commands and hundreds of subcommands each with multiple option flags, and probably hundreds more config options. Gem/Bundle is similar, etc.

If anything, Python is trying to catch up in how complex it’s package managers can be.

Python only has venvs because it needs venvs.

For most other language-provided package-managers the software project you’re working on is the env, so you don’t need to construct or manage a venv at all.

So my point still stands.

> > pipenv allows you to create, manage virtual environments(using venv) apart from managing requirements file(Pipfile) and an npm like locking mechanism, dependency graphs, dev dependencies and more.

> Why on earth would anyone need all this to manage packages for their projects?

> Are there any other programming languages whose package-management comes close to this level of intricacy and complexity?

> “This project needs package X”

> I mean how hard can that be to get right in a self-contained environment?

Packages often have subdependencies and their requirements at times may conflict. If you are very specific in the versions you want, it is more likely to cause issues in dependency resolution.

I would hardly call a dev's machine a "self-contained environment". Most developers I know work in a number of repos with varying requirements, and polluting their system libraries and packages with each project's requirements quickly pollutes the system and can lead to issues.

> I would hardly call a dev's machine a "self-contained environment"

But a software project is.

Only that software project needs those packages.

>“This project needs package X”

>I mean how hard can that be to get right in a self-contained environment?

Okay, I’ll bite.

What version of package X does it need?

Is there a specific version that’s been tested with this project and is known working?

Are there specific versions of its dependencies that have been tested and are known working?

Is it needed at runtime or only at build-time?

What repository can it be found in? PyPI is not the only Python repository; private repos are common.

And as a bonus cherry on top: how easy is it to make sure you have all the project’s dependencies installed in the venv for that project, and that you don’t have packages you’re not keeping track of? This is a UX thing, but developers are human and it matters.

I don't get it either... nor do I understand why venv is considered difficult to use.

"It automatically creates and manages a virtualenv for your projects, as well as adds/removes packages from your Pipfile as you install/uninstall packages."

Well, thanks but automate "python -m venv myvenv" ? Add/remove packages from a "Pipfile" ? Do I have to specify dependencies somwhere else than requirements.txt ? Why ?

There must be some use cases I'm not aware of.

It's about dependencies of dependencies.

Requirements.txt only lists versions of your project's requirements, but Pip actually automatically installs dependencies of those requirements too. And those versions aren't listed in your requirements.txt.

https://realpython.com/pipenv-guide/#dependency-management-w... -- This page about Pipenv vs pip + virtualenv goes into more detail.

You can just 'pip freeze > requirements.txt'

Boom, all recursive dependencies frozen to their current state.

But you don't want to necessarily freeze them to their current state. Yes for repeatable builds, but not in your list of dependencies. They're different concerns and Pipenv splits them.

Giving a shout out to pip-tools here... it's a great, simple solution for pinning your requirements (including the deps of your deps), and keeping your venv's in sync.

pip freeze gets everything installed.

Yes, that 'freezes' everything installed... but that's the point. If you want to update a direct dependency, you just do pip install -u <my-dep>. Any indirect dependencies are updated, and you freeze again.

Sometimes direct-dependency-a and direct-dependency-b have a conflict on which version of indirect-dependency-y you need. This is what we call doing 'actual work.'

^This. I don't understand everyone in this thread complaining that it's hard to update a direct dependency. You literally just pip install it and "pip freeze > requirements.txt" again.

In my experience, the issues come around when you try to build envs cross-platform. There are a lot of dependencies that have missing versions or bugs for certain platforms. This is not a pip/venv problem though--it is more of a python problem.

It’s easy to also get the versions of all sub dependencies and put them in requirements.txt

That makes updating your direct dependencies harder. If you upgrade to using newer version of something, and that things dependencies have changed, you have to manually figure out which dependencies in requirements.txt were for it, and update or remove those etc.

> It’s easy to also get the versions of all sub dependencies and put them in requirements.txt

Is it easy to automatically do this when you want a new package version, without having to remember to do it? Is it easier to put together this process and train other developers in it and be diligent in its use? Is all of that easier than installing an application that has a similar interface to other tools, that does all that for you, and has a community of people to help with issues?

But then you're suddenly responsible for keeping track of your subdependencies and updating the versions of each that you want. That should be up to the dependencies.

Also if you manage to drop a dependency, you don't have an easy way to remove the things from requirements.txt that are only there because they're a subdependency.

Putting it all in one requirements.txt is just too simplistic.

If I had such need, I guess I could have two versions of the requirements.txt:

- One with direct dependencies (versions pinned) - One with direct dependencies + subdependencies (pip freeze output)

Am I being too naive ? (obviously yes if such tool as pipenv exists, but I'm trying to figure why people need *.lock files).

This is close to what pipenv does.

It also adds dependency management. If one subdependency is library_a > 1.0, and another is library_b < 2.0 while also e.g. 2.1 exists, then it will try to find a version between 1.0 and 2.0. Pip doesn't do that.

So in my mind, that's what pipenv is -- pip, virtualenv, those two files, plus dependency management.

> One with direct dependencies + subdependencies (pip freeze output)

That's a lock file...

Ok ! That's what I'd do then.

Are you only working on small projects? For reference, if your project is under 100k lines of code, it's small.

By your arbitrary metric, I work on both small and big projects. Not sure how the loc metric relates to the matter though.

Thanks for the link, interesting read.

I still think this is all too convoluted though. I hope an accepted de facto standard for this will emerge at some point.

1. Because pipenv is easier to use.

2. Because it's not at all clear that pipenv is a third party library. It's made by the same group that makes pip, so it's confusing that pip would be considered a de-facto standard but not pipenv when it's made by the same group, and under the same project in Github.

pip is not yet in the standard library. as opposed to venv.

Whilst you might have to define "in the standard library", pip is added in PEP 453 [0] (accepted in 2013), in a similar capacity to venv:

> However, to avoid recommending a tool that CPython does not provide, it is further proposed that the pip [18] package manager be made available by default when installing CPython 3.4 or later and when creating virtual environments using the standard library's venv module via the pyvenv command line utility.

[0] https://www.python.org/dev/peps/pep-0453/

Yes. And that is confusing, as pip is a de facto standard IMO.

Isn't pipenv a dependency management system that includes the python distribution by way of managing a venv, rather than a simple replacement for venv, sort of like poetry (though poetry, I think, does a better job, but for the problem that it doesn't seem to respect SSL options available for pip which are often needed in enterprise environments)?

pipenv does more than just create a venv, although it is my favorite tool for that. The most important thing it does is freeze the dependency tree using Pipfile.lock

> it does is freeze the dependency tree using Pipfile.lock

sorry but what does freezing the dependency tree mean?

The idea is to make builds (more) reproducible. I can build a python program, test it thoroughly, and then be reasonably assured the whole thing won't come crashing down in CI/CD from a bad update to a transient dependency. Then when I want to update the libraries I know I'm doing it purposefully and can commit the new dependency tree to source control.

probably recording exact dependency versions, based on a loose requirements.txt and when it was built.

You may want this because you have a library that you shouldn't be pinning to the third decimal on a sem-ver package, but that you don't want to hiccup in CI due to a dot-release.

Or maybe you think a loose file your tooling can read, and a hyper-specific file your builder should read, is a better interface for a project.

Yes, kind of like that. Except that it doesn't use requirements.txt but rather a file called Pipfile. In there you can also pin version, or leave them unspecified or only partially specified and you can also divide them in dev-packages and normal packages (so it allows for a bit more flexibility than a requirements.txt file).

a bit like "pip freeze > requirements.txt" then?

> a bit like "pip freeze > requirements.txt" then?

With the added bonus that it also contains a hash of the package so if someone pushes a new version with the same version number it would complain that the hashes don't match.

The Python Packaging Authority declared it the future of dependency management once upon a time and it nominally checked some important boxes such as managing a lockfile.

pipenv install --dev is one reason I guess and pipenv uninstall x gets rid of all the dependencies, that's nice to have