As a Python infra person, I would refine that to:

If you're building an application, use Poetry. If you're building a library, use Flit. Use PEP621 metadata in pyproject.toml regardless.

Poetry is much more focused on managing dependencies for applications than dealing with libraries that have to be used by other libraries or applications. See this deep discussion for some timely/relevant examples: https://iscinumpy.dev/post/bound-version-constraints/

> Poetry is much more focused on managing dependencies for applications than dealing with libraries

The linked article is quite long. Do you have a quick example? I've built lots of apps and libraries using Poetry and haven't run into any dependency issues, and you can specify versions constraints in the same way as Flit, so it's hard to see how they'd be different in that regard.

Edit: I think I see what the issue is--Poetry sets upper version bounds by default via its ^X.Y.Z and ~X.Y.Z syntax and advocates for always specifying upper bounds in their docs.

So, it's not that Poetry can't be used for a library, but you'll have to know that you can, and perhaps should, use different version syntax in some cases (i.e., the standard >=X.Y syntax).

Can we mix Poetry and PEP621 though?

I tried to use project.dependencies and project.optional-dependencies fields in pyproject.toml defined by PEP621. However Poetry seems to ignore it and it tries to stick with its own tools.poetry section.

Ah, you're right. I thought Poetry had adopted PEP 621 already, but it looks like there's still a debate about if/how they want to do that. A bit ironic considering the author of Poetry was listed as an author on PEP 621, and that was accepted just over a year ago.

It went final this March though. So Poetry might needs more time to think about the transitioning plan.

Python packaging is again in a chaos since 2021 because of the rise of standardized PEPs.

I know you didn't mean it that way, but "since 2021"..? :D

I was working actively with Python (and loving it) between 2012-2015 (during which the 2 vs 3 transition/conflicts were raging) and the couple of times I've tried to get back into the groove since then (in side projects) I keep hitting walls and getting confused about how to even get the basic things running.

I've heard good things about Poetry but never tried it. PDM being closer to npm/yarn sounds like an attractive option (since they're very familar), just to tinker around (not to build a complex application with specific requirements). Will give it a shot!

I've seen others make the same reference, but I've not understood it.

My package is a Python module and a set of command-line tools based on that module.

Is my package a library? Or an application? Because it seems to be "both".

It also includes several C extensions, using either the Python/C API manually or Cython. I would love to switch to something more modern than setup.py, but none of these new back-ends seem to support this. For example, I just checked - flit explicitly focuses on "pure Python packages with no build steps" "and leaves the hard things up to other tools".

Do you expect other people to add your project as a dependency of their projects, and call into your API? Then it's a library.

If the only expected users of your API is your own project, it's an application.

And yes, if you need to build native extensions, setuptools is still your only real option. It's not great, and results in hundreds of packages that all find their own definition of insanity. But given just how crazy the C/C++ toolchain ecosystem is, especially when you have to deal with Windows, macOS, and the diversity of Linux, no one else has been will to tackle that problem because setuptools already exists.

Following that definition, the project I mentioned is a library.

But quoting PEP 517 "Much experience suggests that large successful projects often originate as quick hacks".

I have another project which started as a command-line tool only, meant for the end-user to build, maintain, search, and update a data set. It was not designed to be called by another program, neither through import nor popen. Following the guideline, I should use poetry.

It then acquired a limited Python API so search functionality could be incorporated in a web app. One of my users paid for that feature. I don't know if others use it.

This turned my package into a library. Does that mean I should switch from poetry to flit?

There seem to be several people who have tackled that problem, including emscons, PyQt-builder, and mesonpep517, which all defer to an existing third-party build system. I don't use or have experience with any of the underlying build systems, and version numbers like 0.28.0 and 0.2 make me wary.

Are there others? Finding such modules on PyPI is not easy! A search for "517" is almost useless. Could there be a new trove category for such packages?

Sold everyone on using poetry a few months ago and now red faced as we have a litany of problems and time waste due to it. We are now sitting on some bleeding edge branch because specific dependencies cannot work at all without some new fangled feature and everyone wishes we were just using plain virtualenv as we had much less problems with that.

You would also not have any real control over what gets installed with just venv. So everyone might think the grass was greener on the other side, but it might not actually be, if you value reproducible environments.

What you've never spent hours trying out n! orderings of requirements.txt until you get the results you want/can live with? You have not lived until you've done this. /s

Dep Hell is real.

In other words, you work with people who have no clue what’s going on.

Thanks, that's good advice!

Great. I have heard "anecdata evidence" that sometimes poetry fails to install a combination of packages or something along those lines, did you find any of those shenanigans in your own experience?

(Not GP) Occasionally, but still use poetry.

The issues happen because poetry is strict about version constraints. Pip used to be lax, so some packages could get away with poor/overly restrictive constraints..

Now pip is strict as well, and poetry is also getting a lot more common, so maintainers have had pressure to fix most of these issues.

I've run into this a couple of times. Typically it's when starting a new project and gradually adding dependencies. By default, when adding (poetry add <dep>), poetry adds the most recent version of the library, and in pyproject.toml that becomes the minimum possible version. If you then later try to add something that depends on that library, but isn't up-to-date on the most recent version, poetry will (correctly!) error out because it fails to find a compatible solution. For example, something like this:

    poetry add ingredients  # most recent version: 1.0
    poetry add cake         # requires ingredients, but <1.0

So here, poetry would add ingredients to your pyproject.toml with version = "^1.0.0" in the first step. In the second step, when you tried to add cake as a new dependency, it would say it can't find a version of cake compatible with your pyproject.toml's requirement for ingredients.

The solution is to change pyproject.toml to be more permissive about the version of ingredients, resulting in it downgrading to a compatible version. Or submit an issue and/or PR with the maintainers of cake to bump up the supported versions.

In PDM, add command supports a `--unconstrained` option to relax the version constraints to find a solution. You don't need to edit the file manually.

Another way it happens: I was stuck on python 3.6 for a while and it meant that I couldn't just poetry add pandas because the latest version isn't compatible with 3.6 and it won't look for other versions.

No shenanigans for any library-ish code with few dependencies that targets relatively modern Python.

I've had one or two fundamental version conflicts with a 5+ year old application with 100+ dependencies and a decent amount of legacy stuff. They were a pain in the ass, and the sdispater's stance on not allowing overrides is a pain in the ass. We ended up forking the upstream libraries to resolve the version conflict.

With all of that, poetry is amazing and a huge step forward. I'd advocate it wholeheartedly.

I've spent the last year or so comparing these packages. Initially, I liked poetry, but I began to run into issues related to client cert auth with private repositories. I and several other users who encountered these problems submitted pull requests to address them, but found that it was nearly impossible to get any of them reviewed, much less merged.

Out of frustration, I decided to try pdm instead, and although pdm doesn't handle pushing to pypi repositories, Twine works fine for it, and I have had a pain-free experience using pdm for dependency management.

I like the idea of poetry, but it's unusable if I can't get bugs fixed, and I gave up after nearly a year of trying.

> but I began to run into issues related to client cert auth with private repositories.

Yeah, I think that's the worst thing about private infrastructure in 2021 is getting those certs correctly installed and seen by the software and both sides.

I'm thinking that tailscale or something like it may be a better alternative in the future where authorization is at the link layer before the tcp connect can happen.

As others have said, this is related to how Poetry does dependency resolution. If you say your program uses Python "^3.8", you would expect any Python version in that range to work, but instead, ALL Python versions in that range must work.

If one of your dependencies won't work with 3.9, the dependency graph cannot resolve. The author of Poetry said that this is because Poetry is meant for libraries, where this makes more sense.

Except Poetry's recommendations/decisions around version capping doesn't make sense for libraries. https://iscinumpy.dev/post/bound-version-constraints/

Is Poetry used a lot by Python devs? What package management a lot of Python devs use?

I mostly code on JavaScript and (obviously) use NPM a lot, and it makes me wonder.

I've been using Poetry exclusively for the past few years. I had a couple small problems with it early on, but lately I haven't run into any issues.

I use it for all of my projects, yes. I haven't really had many problems, just one or two that were easily resolved.