1) It’s batteries included. The standard library is a core feature, and it is extensive in order to reduce the number 3rd party library dependencies.
2) “There should be one— and preferably only one —obvious way to do it. Although that way may not be obvious at first unless you're Dutch.” The Pythonic way is to use the standard library, setup.py, pypi, and so on. Python is not Rust, or Node, or Clojure, or anything else. Do it Pythonicly and good things will happen.
In the history of Python, it’s only because of the Zen’s guiding principles and the “come join us” attitude of the community that Python accidentally became “enterprise grade”. It succeeded in spite of the lack of Cargo or NPM, or (shudder) Mavin[, or CI/CD, or 3D game engines, or easy binary package productization, or everything else Python is obviously bad at]; almost precisely because it’s not those things. Python encourages me to get my task done, and all it asks is that I leave the flame wars at the door and embrace it for what it is.
It saddens me to see so many people jump down zomglings throat for making the daring suggestion that Pythonistas use the language as designed. Because, he is absolutely right about this: “Complex is better than complicated.”
Like it or not Python is now “enterprise grade”. The sooner it either grows up and figures out packages, ci/cd, binaries etc, or it looses its reputation at a great beginner and general programming language to something like Julia or Crystal, the better.
In the python ecosystem, poorly defined packages is a wide problem. You never know what you get in the next version upgrade.
So my suggestion: burn all the "simple" python packaging tools in a big fire and move eveything to Poetry today. It will be painful at start but will increase the quality of the ecosystem by a huge margin if just 50% of the projects do this.
Rust is incredibly lucky to have been created when it was because it benefits immensely from these things (I think it's the best new-to-middle-aged production-usable systems language out there today).
I agree with the idea but languages like Python and their ecosystems are really hard to move (remember python 2->3? is that even over?) -- it's a herculean and often impossible task.
A lot of ecosystems put on some solid horse blinders in order to avoid Java at all costs (Javascript/Node/NPM being another example). They've avoided good ideas from Java for more than a decade.
If poetry create consume non-poetry packages and create packages which other package management systems can consume? If so, then:
1. Projects can move more effectively independently.
2. Projects can re-package as a single task rather than a massive rewrite effort.
https://blog.jayway.com/2019/12/28/pyenv-poetry-saviours-in-...
It's the best one I've tried so far.
* virtual environments (granted, venv is now part of the stdlib, but it's still an "extra tool")
* publishing to PyPI or another index (twine)
* dependency management (both for development and actual dependencies)
* unit tests (pytests)
* static typing (mypy)
* linting (flake8, pylint)
* styling (black)
Finally, tools like Poetry [2] offer a whole set of functionalities in one place (dependency management, virtual environments, publishing), which means that they need fewer "extra tools" than just setuptools.
[0] https://www.python.org/dev/peps/pep-0518/
[1] https://setuptools.readthedocs.io/en/latest/build_meta.html
poetry build -f wheel
poetry publish
Sure, experts know how to do all these things because they spent many days learning them, but I'd rather outsource to a tool.
Version pinning can be done in setup.py using the same syntax you would see in a requirements.txt file. You should be very conservative when pinning versions in a library, though.
You can lean on your ci tool (eg. Github actions) to handle testing, hash checking, credential management, etc. But I recommend all of this start as a bunch of locally runnable scripts.
I typically bump version directly in a version file and move on with my life.
This stuff usually builds up iteratively and at least for me has never been the starting point. Starting point should be a library worth sharing. It is not the end of the world of you release the first few versions manually.
Even if that set of tools is kinda crappy (Glances at go), it's just so nice to not have all the bikeshedding and just get on with it.
E.g. more flexible than Cargo in that you could have a large codebase with two different versions of a dependency isolated to their compile units allowing you to gradually adopt a breaking change dependency in a large code base. I was kinda bowled over with that feature (the isolation part is key).
For python I’m finding poetry much more ergonomic than pipenv, it’s not just the speed difference it’s the convenience of one tool which aligns with what you’re saying although the existence of poetry doesn’t delete historical aberrations in Python’s long history.
What we saw with npm was the entire community iterating towards a feature set and everyone reaping the benefits automatically with npm updates. package-lock.json is a good example of this.
Leiningen is the weird one where people did actually settle fairly well around an unofficial solution in the absence of an official one. I think the norm with languages that forego official tooling is closer to what we've seen in Python.
If you're curious, the email threads about Conda and defining wheels are interesting.
[1]: https://github.com/pypa/pipenv
But with or without a BDFL, one packaging tool to rule them all is a pretty tall order. The needs of a package like scipy, which incorporates C & Fortran code, are pretty different from something like requests. And different communities using Python have their own entrenched tools and techniques. It takes more than someone saying "Foo is officially blessed" to shift that, even if everyone respects the person saying that.
Also, yes, Guido and many other early contributers have been been active for 30ish years.
And NumPy/Scipy/Pandas exist (and are successful) because...? /s
The js tooling is particularly immature, e.g. fake packages along with typo squatting is rife. Compare with boring old maven where that’s not been an issue in >10 years at this point.
Personally I much prefer pip over npm, it feels much more polished.
python is a mature, old, software system. three times older than go or rust. way older than zig. these modern languages have learned from the field as a whole and implemented tools that people are taking for granted nowadays.
as with any mature software system, people & companies have established their preferred ways of doing things. it's going to be hard to have the language council dictate ways of doing things.
i would recommend going simple and using the standard set of modules until poetry or whatever gets into it.
Strongly disagree. There are so many footguns with low-level tools like pip that I can't recommend it to anybody but an expert (but an expert doesn't need my recommendation anyway).
It's the succeeding sections (A, B, C, D, E) that get more advanced, but they're all optional. You should definitely do A, but the rest I'd say it's a lot more opinionated and definitely not needed.
The problem with this approach is that it doesn't handle transitive dependencies well. Say you depend on version 1.4.6 of a particular library. And then that library depends on version >= 2 of some other library. When you install your package, you know that you'll get version 1.4.6 of the first library but have no idea what version you'll get of the second library. You can of course pin all the transitive dependencies - except that clutters up the setup.py and is a massive pain to keep up to date as you bump individual dependency versions.
the version selected during a build is the one with the most minimal version that satisfies all other constraints. this means if you have libA that needs dep>=1.1 and libB that needs dep>=1.3, you get dep=1.3 even if dep1.9 is out. your build never changes because of a new version release, as long as they release with proper semantic versioning. if you later include libC that needs dep>=1.8, you'll get that version. but because you changed your immediate dependencies, not due to a surprise down the dependency line.
Imagine that you depend on library A of a particular version which itself depends on library B. With minimal version selection, as long you don't bump you dependency on library A (or some other library that depends on library B), you'll continue to get that same version of library B. But, then library B releases a critical security fix. With minimal version selection, there isn't a great way to pick up that fix. You can _hope_ that library A releases a new version that requires the fix - but that may or may not happen and could take a while. Or, you could add an explicit dependency on the new version of library B - which is unfortunate, since, your main package doesn't depend on library B directly.
Lock files solve this problem. You can depend on whatever version of library A that you need and lock the transitive dependencies. And once library B releases its fix, you can update your lock file without having up bump the version of library A.
Tools like poetry providing the features to automate this workflow.
No; you should be very conservative when pinning versions in an application, not in a library. Check this article for the explanation: https://caremad.io/posts/2013/07/setup-vs-requirement/
Why spend time tweaking the setup when you can just get it right in the first place with less work?
I've been meaning to try it but haven't had the time to migrate an existing project to poetry.
poetry new foo
There is still room for improvement, e.g. I had some trouble with their install script at times (python vs python3)
If you're an expert, you can do it with setup.py or setup.cfg, but I don't think it's normally worth the trouble.
You seem to be confusing packages with "apps". It's very important to understand the clear distinction between these.
I'm not confusing libraries with applications. Pinning dependency versions enables a repeatable test environment.
It's by far the simplest option I've found.
If your project is a library, just use setup.py and express your deps abstractly (min, max or version range). Don't pin to specific versions at all, if you can help it.
The author is correct that you want a tool such as flit or poetry which will work with pyproject.toml. Setting up a basic package will be no harder than using setuptools, and it is much more future-proof. You won't have to copy-paste some other crufty setup config either.
It is fair that you don't need all the external tools in this tutorial. In particular, using make is very silly since you can configure different linting and testing workflows directly in pyproject.toml, rather than pull in a whole other system which only works decently on *nix. Poetry also removes the need for tox.
It does not correctly verify hashes (if at all) [1]. You can't add packages without updating all your dependencies. Monorepos are not supported. PEP-508-compliant Git dependencies cause the tool to crash with TypeError [2].
I think Poetry is the right direction, I use it for everything, but it's not the silver bullet you're painting it to be (yet). It's definitely not on par with Cargo, or maybe even npm.
[1]: https://github.com/python-poetry/poetry/issues/3765 [2]: https://github.com/python-poetry/poetry/issues/3425
I have to say that it probably isn't a fare comparison because python is much older than nodejs. Python package management might very well have been state of the art in 1995.
This approach ends up with multiple, potentially-incompatible versions of the same package in a project. True that's less of a problem in JS, since it's interpreted (deferring imports to runtime) and un(i)typed (no need to check if interfaces match up). Yet even that has lead to replacements/complements like yarn.
Yes. There's hardly even a standard directory structure, let alone a standard way to convert source code to published code. Every slightly non-trivial repo basically has an ad hoc build system of its own. Ever tried to fix a bug in a package, and realized that using git://github.com/user/repo#branch doesn't work, because npm only downloads the source code, which bears no resemblance to the built products? I fixed two bugs in two third party packages within the past week, had to deal with this twice. Ran into the Node 12+ and Gulp 3.x incompatibility issue twice in the past month (with totally modern, actively developed packages), too.
npm has more sophisticated dependency resolution and locking than pip, sure. Python packaging is more consistent in basically every other regard.
If data generated by one version of a library is being consumed by another version, that's a bug in the code that moves data between them.
Are you writing a simple library? Create a setup.py. Copy paste an existing setup.py and modify it to suit your purposes. Now you have a working, pip installable python package.
Want to publish to PyPI? Use twine. It's standard and it's simple.
You don't need complicated tooling for simple projects.