Re:Thoughts on the Python Packaging Ecosystem
by MrJohz on Saturday January 21, 2023 @11:44PM
(#34474377)
I've heard this claim often, but only ever from people using worse package managers than NPM, so maybe I just don't know what people want in a package manager.
Things NPM has by default that are impossible to achieve in Python-land without external tools:
* Lock files. If you check in `package-lock.json` (which is always automatically generated), then everyone who clones and installs your project will install exactly the same versions, every time. In Python you need to either run pip freeze manually when you make a change (and even then, that's not quite the same thing), or you need to use third party tools with varying degrees of manual labour.
* Local installs. Unless you explicitly ask for it, NPM will always try and set up an isolated, local environment for every project you work on. I don't think this is even possible with pip alone, which is why Python also now comes with a sledgehammer of a solution: venvs.
* Consistency. `npm install` will set the project up. `npm run <TAB>` will give me a list of useful development commands. Packages are always installed to `node_modules`. Most NPM projects work in pretty much the same way, which makes it very easy to make changes to other people's projects, or get started on your own. I have very rarely seen two Python packages that work the same. Different venv names, different places where the dependencies are specified, different procedures for installing those dependencies, different ways of locking dependencies (if any were used at all), etc, etc. That's not too get started on the different options for task runners.
* Workspaces. It turns out, it's often useful to develop multiple Python packages at the same time (because they need to be consumed by different people or different systems, or just because it's convenient to split a project up sometimes). With NPM you just need to configure this once and it'll all get linked up properly by default. How do you do this in Python? I ended up leaving my last job before I managed to figure this out, which was disappointing to my sense of professional pride, but oh so valuable for my sanity. Now I work with Javascript projects and don't need to worry about things like that because it just works.
* Multiple versions of the same package can be installed at the same time. Wasteful? Possibly. Useful? Very. When a dependency makes a breaking change, it doesn't need to split the whole ecosystem in two while we wait for all of the other dependencies to update their version. This is simply impossible with Python.
And that's just a comparison with Python, which at least has a package manager. I've heard the same claims from people who develop C/C++ projects, which can surely only be a form of Stockholm syndrome...
Things NPM has by default that are impossible to achieve in Python-land without external tools:
* Lock files. If you check in `package-lock.json` (which is always automatically generated), then everyone who clones and installs your project will install exactly the same versions, every time. In Python you need to either run pip freeze manually when you make a change (and even then, that's not quite the same thing), or you need to use third party tools with varying degrees of manual labour.
* Local installs. Unless you explicitly ask for it, NPM will always try and set up an isolated, local environment for every project you work on. I don't think this is even possible with pip alone, which is why Python also now comes with a sledgehammer of a solution: venvs.
* Consistency. `npm install` will set the project up. `npm run <TAB>` will give me a list of useful development commands. Packages are always installed to `node_modules`. Most NPM projects work in pretty much the same way, which makes it very easy to make changes to other people's projects, or get started on your own. I have very rarely seen two Python packages that work the same. Different venv names, different places where the dependencies are specified, different procedures for installing those dependencies, different ways of locking dependencies (if any were used at all), etc, etc. That's not too get started on the different options for task runners.
* Workspaces. It turns out, it's often useful to develop multiple Python packages at the same time (because they need to be consumed by different people or different systems, or just because it's convenient to split a project up sometimes). With NPM you just need to configure this once and it'll all get linked up properly by default. How do you do this in Python? I ended up leaving my last job before I managed to figure this out, which was disappointing to my sense of professional pride, but oh so valuable for my sanity. Now I work with Javascript projects and don't need to worry about things like that because it just works.
* Multiple versions of the same package can be installed at the same time. Wasteful? Possibly. Useful? Very. When a dependency makes a breaking change, it doesn't need to split the whole ecosystem in two while we wait for all of the other dependencies to update their version. This is simply impossible with Python.
And that's just a comparison with Python, which at least has a package manager. I've heard the same claims from people who develop C/C++ projects, which can surely only be a form of Stockholm syndrome...