News for hackers. Stuff that matters.
Re:How to improve Python packaging, or why 14 tools are at l... by woodruffw on Sunday January 15, 2023 @01:01PM (#34391429)
It's not very well documented, but the PyPA tools do provide a unified experience when used correctly.

Here's a PyPA project (FD: one I work on) that uses a single pyproject.toml to handle all aspects of packaging (and most non-packaging tool configuration, to boot)[1]. With a single file like that, the only thing you need to do to start a local development environment is:

    python -m venv env && . env/bin/activate
    python -m pip install .[dev]
(We provide a `make dev` target that does that for you as well.)

Similarly, to build distributions, all you need is `build`:

    python -m build
[1]: https://github.com/pypa/pip-audit
Re:How to improve Python packaging, or why 14 tools are at l... by qbasic_forever on Sunday January 15, 2023 @01:11PM (#34391509)
Nowhere in pypa documentation is your simple workflow described or mentioned. Instead it's a jumble of links to a myriad of tools including hatchling, flit, pdm, etc. and basically just a shoulder shrug, 'I don't know, figure it all out yourself' message. This article makes a great point that the current pypa 'guidance' is too confusing and vague for actual end users (i.e. people that don't work directly on pypa or haven't been studying python packaging for decades).
Re:How to improve Python packaging, or why 14 tools are at l... by woodruffw on Sunday January 15, 2023 @01:18PM (#34391589)
I agree that it's confusing. That being said, there is an official PyPA tutorial that goes through the exact steps need to produce the commands I suggested, so I don't think it's accurate to say that it's nowhere to be found[1].

Edit: Unfortunately it's easy to confuse the above tutorial with this one[2], which is specifically for setuptools. So I can appreciate end user confusion around the documentation, particularly in that respect!

[1]: https://packaging.python.org/en/latest/tutorials/packaging-p...

[2]: https://packaging.python.org/en/latest/guides/distributing-p...

Re:How to improve Python packaging, or why 14 tools are at l... by qbasic_forever on Sunday January 15, 2023 @02:11PM (#34392128)
No I'm looking at link 1 and specifically have an issue with the huge table of options for how to setup your pyproject.toml--it has tabs for hatchling, flit, pdm, etc. but _zero_ description of why I would want to use one of those tools. It's just more confusing, like you're being thrown a bag of parts and no description of what to build or how to do it.

To be honest that entire pypa doc should be like two paragraphs long instead of 1000+ words. It should be basically, "ok to package your python app just run <official python tool to init a package>, that's it you're done!". Every decision should be made for me, like it is with npm, cargo, etc. I shouldn't have to think beyond running one command.

That's what python end users need. We don't need a million tools and huge docs with seemingly no vision or goal. We need to get shit done and the less time we faff about with packaging and tooling the better.

Re:How to improve Python packaging, or why 14 tools are at l... by KyeRussell on Sunday January 15, 2023 @06:42PM (#34394911)
Yep. There being no documentation to really empower someone to work out what to use, to me very directly says “there’s some political spat going on that I’m being exposed to here”.
Re:How to improve Python packaging, or why 14 tools are at l... by theacodes on Sunday January 15, 2023 @10:51PM (#34396582)
that "political spat" was actually targeted harassment towards a former PyPA member the last time someone actively working on packaging.python.org tried to be remotely opinionated. Since this ended with them stepping away and no one else has volunteered, it remains unopinionated.
Re:How to improve Python packaging, or why 14 tools are at l... by qbasic_forever on Monday January 16, 2023 @12:41AM (#34397223)
Sounds like the steering council or whatever that runs python nowadays needs to step up. If people are being targeted for trying to improve packaging then the steering council should make a decision and direct all commentary, etc. to them, not individuals documenting best practices.

The whole 'let the community figure it out' seems to have failed and is causing nothing but more confusion and now attacks on people. They need to step up and say, "this is how python packaging will work, period. End of story, end of debate.". That was the one good thing a BDFL model for leadership could achieve, making a hard decision.

Re:How to improve Python packaging, or why 14 tools are at l... by woodruffw on Sunday January 15, 2023 @10:52PM (#34396592)
Edit: My sibling comment has another answer, which makes me believe I'm lacking context. So I've removed this comment, in the interest of not offering opinions outside of what I know.
Re:How to improve Python packaging, or why 14 tools are at l... by Izkata on Sunday January 15, 2023 @01:44PM (#34391847)
AFAIK in the python 2 era it was the de-facto standard (just swap "python -m venv" with "virtualenv"). All these new tools have just made it more complicated and it's not clear to me what they gain, seems to me like they've simply succeeded in convincing people it's complicated by obscuring what's actually going on.
Re:How to improve Python packaging, or why 14 tools are at l... by vonseel on Tuesday January 17, 2023 @01:45PM (#34416784)
Correct. And most people back then were just using `pip install -r requirements.txt`, including different files for dev/test/prod as needed.

I worked at one company that used buildout back then, which seemed relatively rare, but buildout was more flexible and allowed you to do things like installing system packages or running your own scripts. We used it to pull down config files that weren't checked into the repo but still wanted to be shared, helping setup the local developer environment, and a few other things.

Re:How to improve Python packaging, or why 14 tools are at l... by fbdab103 on Sunday January 15, 2023 @05:04PM (#34394070)
A few weeks ago, I was attempting to help introduce someone to Python. I felt embarrassed at trying to paper-over how complicated and confusing the initial configuration is for a newbie. No officially endorsed best-practices to which I could point. Half a dozen tribes with their own competing solutions and guidance.

A definitive statement from Python.org as to The Way would go so far.

Re:How to improve Python packaging, or why 14 tools are at l... by Denvercoder9 on Sunday January 15, 2023 @01:28PM (#34391687)
That repository doesn't seem to pin dependency versions. How do you integrate that in this workflow?
Re:How to improve Python packaging, or why 14 tools are at l... by woodruffw on Sunday January 15, 2023 @01:35PM (#34391752)
You could use `pip-compile` if you want full pinning. That's what we do on another project -- we use GitHub Actions with `pip-compile` to provide a fully frozen copy of the dependency tree for users who'd like that[1].

In the context of `pip-audit`, that makes a little less sense: most of our dependencies are semantically versioned, and we'd rather users receive patches and fixes to our subdependencies automatically, rather than having to wait for us to release a corresponding fix version. Similarly, we expect users to install `pip-audit` into pre-existing virtual environments, meaning that excessive pinning will produce overly conservative dependency conflict errors.

[1]: https://github.com/sigstore/sigstore-python/tree/main/instal...

Re:How to improve Python packaging, or why 14 tools are at l... by Izkata on Sunday January 15, 2023 @01:47PM (#34391864)
Or if you don't want to install something else and are willing to just use version numbers (instead of also hashes like pip-compile in that link), "pip freeze" is built in.
Re:How to improve Python packaging, or why 14 tools are at l... by woodruffw on Sunday January 15, 2023 @02:05PM (#34392067)
The tricky thing with `pip freeze` is that it dumps your environment, not your resolved set: your environment also contains things like your `pip` and `setuptools` versions, any developing tooling you have, and potentially your global environmental state (if the environment has global access and you forget to pass `--local` to `pip freeze`).

In other words, it's generally a superset of the resolutions collected by `pip-compile`. This may or may not be what you want, or what your users expect!

Re:How to improve Python packaging, or why 14 tools are at l... by Izkata on Sunday January 15, 2023 @02:16PM (#34392192)
By default (at least in python3 nowadays) it also excludes pip, setuptools, distribute, and wheel; you need "--all" to include them in the output.
Re:How to improve Python packaging, or why 14 tools are at l... by woodruffw on Sunday January 15, 2023 @02:19PM (#34392226)
Oh, that's news to me! I stand corrected.

(The point about other development tooling is, I believe, still accurate -- if you e.g. have `black` installed, `pip freeze` will show it.)

Re:How to improve Python packaging, or why 14 tools are at l... by fbdab103 on Sunday January 15, 2023 @05:08PM (#34394109)
Which is a huge limitation of many of the other tools. I have some beef with poetry, but it did at least get one thing correct: differentiating between the code required libraries and the development tooling (pytest, black, etc). There are hacky workarounds to accomplish this with other tools, but codifying this differentiation is incredibly valuable.
Re:How to improve Python packaging, or why 14 tools are at l... by vladvasiliu on Monday January 16, 2023 @01:28AM (#34397511)
With pip-tools you can use a requirements.txt for production and a requirements-dev.txt for your development environment. The latter imports the former.
Re:How to improve Python packaging, or why 14 tools are at l... by samwillis on Sunday January 15, 2023 @03:11PM (#34392814)
Ultimately it needs to be "Python.org" that endorses the tool, not PyPA, no one in the scheme of things know who PyPA are and if it's the "one true way".

If you go to Python.org and follow through to the beginners guide [0] this is what's suggested:

> There are several methods to install additional Python packages:

> Packages can be installed via the standard Python distutils mode (python setup.py install).

> Many packages can also be installed via the setuptools extension or pip wrapper, see https://pip.pypa.io/.

That is so out of date and fundamentally confuses people coming to Python for the first time. How is pip secondary, and no mention of venv?

The PyPA need to get buy in from Python Core to put one tool front and centre first. It needs to be like Rust with Cargo, literally the first thing you learn to use and core to all beginners guides.

That's not to diminish the work of PyPA, you are all amazing I just want your work to be more obvious!

0: https://docs.python.org/3/using/mac.html#installing-addition...

Re:How to improve Python packaging, or why 14 tools are at l... by Godel_unicode on Sunday January 15, 2023 @05:20PM (#34394230)
The future is here, it’s just inconsistently distributed. There’s also the problem of python.org not wanting to pick winners, i.e. not promoting activestate vs conda vs pypa.

https://packaging.python.org/en/latest/tutorials/installing-...

https://packaging.python.org/en/latest/tutorials/packaging-p...

Re:How to improve Python packaging, or why 14 tools are at l... by KyeRussell on Sunday January 15, 2023 @06:39PM (#34394897)
Which is just, utterly utterly ridiculous. The Python packaging story will continue to be a mess right up until this particular inconsistency is resolved.
Re:How to improve Python packaging, or why 14 tools are at l... by agumonkey on Sunday January 15, 2023 @03:11PM (#34392820)
where is the

    install .[dev]
format/syntax defined ? I was trying to find what was possible and how to make sense of it in setup.cfg tools

I found one mention in the docs but no more.

Re:How to improve Python packaging, or why 14 tools are at l... by woodruffw on Sunday January 15, 2023 @03:15PM (#34392861)
It’s a pip-ism; as far as I know, it’s not defined in any PEP. It should be in their documentation, however.
Re:How to improve Python packaging, or why 14 tools are at l... by agumonkey on Sunday January 15, 2023 @05:30PM (#34394314)
gonna dig deeper there then, thanks
Re:How to improve Python packaging, or why 14 tools are at l... by baq on Sunday January 15, 2023 @04:03PM (#34393368)
you may have done

   pip install jupyter[notebook]
this is the same thing, just for . (current directory) and dev variant.
Re:How to improve Python packaging, or why 14 tools are at l... by woodruffw on Sunday January 15, 2023 @04:12PM (#34393469)
Yep. The key difference is that the former is specified in PEP 508, while `.` and its variants are a convenience feature that pip (and maybe some other tools) provide.
Re:How to improve Python packaging, or why 14 tools are at l... by agumonkey on Sunday January 15, 2023 @05:18PM (#34394201)
yeah but I'm missing data on how variant interacts with various requirements defined in setup.cfg (testing dependencies for instance)
Re:How to improve Python packaging, or why 14 tools are at l... by nprescott on Sunday January 15, 2023 @04:28PM (#34393644)
PEP-508[0] explains the grammar for "extras":

> Optional components of a distribution may be specified using the extras field:

  identifier_end = letterOrDigit | (('-' | '_' | '.' )* letterOrDigit)
  identifier    = letterOrDigit identifier_end*
  name          = identifier
  extras_list   = identifier (wsp* ',' wsp* identifier)*
  extras        = '[' wsp* extras_list? wsp* ']'
as well as explaining their behavior, albeit briefly:

> Extras union in the dependencies they define with the dependencies of the distribution they are attached to.

The resolution on . is explained by the pip documentation[1]:

> pip looks for packages in a number of places: on PyPI (if not disabled via --no-index), in the local filesystem, and in any additional repositories specified via --find-links or --index-url. There is no ordering in the locations that are searched. Rather they are all checked, and the “best” match for the requirements (in terms of version number - see PEP 440 for details) is selected.

[0]: https://peps.python.org/pep-0508/#grammar

[1]: https://pip.pypa.io/en/stable/cli/pip_install/#finding-packa...

Re:How to improve Python packaging, or why 14 tools are at l... by lmm on Sunday January 15, 2023 @05:58PM (#34394579)
That's still about 4 times as much command as I'd consider normal/reasonable, compared to say "mvn install".
Re:How to improve Python packaging, or why 14 tools are at l... by Helmut10001 on Monday January 16, 2023 @01:36AM (#34397564)
First time I hear PyPA, and looking up their website [1]: It is not a good sign if a packaging authority has a broken logo link on their mainpage [2].

[1]: https://www.pypa.io/ [2]: https://pypi.org/static/images/logo-large.svg/

Re:How to improve Python packaging, or why 14 tools are at l... by vonseel on Tuesday January 17, 2023 @01:40PM (#34416706)
So, the project you linked has a pyproject.toml file uses flit:

    requires = ["flit_core >=3.2,<4"]
    build-backend = "flit_core.buildapi"
Python.org defaults to using hatchling in the docs[1].

PyPA has a sample repository using setuptools[2]. This project is linked from Python.org on this page.[3]

Python.org recommends using setuptools under packaging recommendations as well, with no mention of hatchling or flit even on that page.[4]

No wonder so many devs are frustrated about this. I've been writing mostly Python for work and fun the past 11 years and I'm f*cking confused what tools to use. When I start a new side project I don't know whether a tool is going to exist and still work in a few years, because there's 10 tools trying to do the same things and it's not clear what the community is actually getting behind.

[1]: https://packaging.python.org/en/latest/tutorials/packaging-p...

[2]: https://github.com/pypa/sampleproject/blob/main/pyproject.to...

[3]: https://packaging.python.org/en/latest/guides/distributing-p...

[4]: https://packaging.python.org/en/latest/guides/tool-recommend...