News for hackers. Stuff that matters.
Re:Boring Python: dependency management (2022) by irjustin on Sunday February 04, 2024 @09:25PM (#39256600)
With the move to pyproject.toml in 2022[0], Poetry has become our goto method.

With the lock file being default, we don't worry about different installs in different envs.

Having come from the Rails world, Bundler system was solved for... a decade? So I was surprised it was such a mess in Python until so recently.

At the core, the thing that makes Poetry and Bundler system so predictable is 1, lock file. 2, ability to install different versions in diff locations and referencing the version you need to load. Each alone isn't enough.

npm had the same problem pip suffered from, you may have a version installed different what the req.txt, proj.js or even lockfile says but because it exists inside the location, it gets loaded. It wasn't until yarn2 did node_modules finally get moved out such that side-by-side versions wasn't awkward.

[EDIT]

If you're not using Poetry + Docker for deployment yet, I 100% recommend it as the "boring" method.

  RUN curl -sSL https://install.python-poetry.org | python -
  RUN /root/.local/bin/poetry config virtualenvs.create false
  
  COPY poetry.lock pyproject.toml ./
  
  RUN --mount=type=secret,id=gh_priv_key,target=/root/.ssh/id_rsa \
      /root/.local/bin/poetry install --without dev --no-root

[0] https://packaging.python.org/en/latest/tutorials/packaging-p...
Re:Boring Python: dependency management (2022) by doctorpangloss on Sunday February 04, 2024 @10:28PM (#39256967)
Despite these comments, Poetry is pretty limited. You cannot install PyTorch with it for example, and the developers do not care: https://github.com/python-poetry/poetry/issues/6409

So basically the single most exciting thing to ever happen to the ecosystem isn’t supported by Poetry.

Re:Boring Python: dependency management (2022) by irjustin on Sunday February 04, 2024 @11:25PM (#39257314)
Based on this comment 5 days ago[0], it's working? I'm not sure didn't dig in too far but based on that comment it seems fair to say that it's not fully Poetry's fault because torch removed hashes (which poetry needs to be effective) for a while only recently adding it back in.

Not sure where I would stand if I fully investigated it tho.

[0] https://github.com/python-poetry/poetry/issues/6409#issuecom...

Re:Boring Python: dependency management (2022) by doctorpangloss on Sunday February 04, 2024 @11:52PM (#39257501)
Every comment in that thread, including the comment you referenced, is wrong.

Due to Poetry's architecture, it can't satisfy all three at the same time:

- the platonic ideal of build isolation and lockfiles

- installing the appropriate accelerator specific version of pytorch for the current platform non-interactively or one that the user selects

- dependencies for pytorch that are transitively compatible with the way dependencies for pytorch are expressed elsewhere

This is something you can achieve with setuptools and setup.py by forfeiting the platonic ideals of build isolation and lockfiles.

Poetry, on the other hand, does not let you choose which lamb to sacrifice. Everyone in the thread, for the last two years, who has reported that they have had some success are misunderstanding the state of their install, and have interacted with flaws in all three situations I'm describing. Usually they have something that will not correctly install anything that is dependent itself on PyTorch, which is essentially useless, since everything in the PyTorch ecosystem is is dependent on it, and the main workaround the community uses - installing torch first, followed by installing dependencies from a requirements.txt, followed by copying a dump of scripts - is not compatible with poetry.

Here's the facts of the matter:

- 1.2M requirements.txts https://github.com/search?q=path%3A%2F%5Erequirements.txt%24...

- 664k setup.pys https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F&type...

- setup.pys that reference requirements.txts https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F+requ... 67.1k

- only 30k pyproject.toml specified with dependencies https://github.com/search?q=path%3A%2F%5Epyproject.toml%24%2...

Basically 2/3rds of Python end users do not engage with packaging at all. pyproject.toml with dependencies is about 1.5% of the ecosystem. It basically provides only downsides compared to setup.py and pinning your package versions by commit in your setup.py dependencies aka doing exactly what golang does, and this does not require any external tools in Python. In my opinion, the Poetry developers need to fix pytorch or they will not get adoption during Peak Python.

Re:Boring Python: dependency management (2022) by irjustin on Sunday February 04, 2024 @11:55PM (#39257515)
Got it, thanks for the detailed update. Clearly much more complex.
Re:Boring Python: dependency management (2022) by ildjarn on Monday February 05, 2024 @02:53AM (#39258426)
Is there a path to declarative package manifests that reach Poetry’s platonic ideals?

Frankly setup.py is a big hack and we should strive to do better.

Re:Boring Python: dependency management (2022) by albertzeyer on Monday February 05, 2024 @03:27AM (#39258620)
Sorry if my questions are maybe not well informed (maybe I should delve more into Poetry, I don't really know it), but I don't really understand those points, but I would like to understand them.

What does it mean, build isolation? And how does PyTorch violate that? Or why can't we install PyTorch with build isolation?

What is the issue with lockfiles?

I can understand there might be issues if you need to install a custom accelerator-specific version of PyTorch (although also I don't fully understand what the issues are about this). However, what's with just the standard version, which is suitable for most people, as it comes with CUDA support?

What do you mean by non-interactively? pip install is non-interactively, or not?

What issues are there with the dependencies for PyTorch? Why are they not compatible with the dependencies expressed elsewhere?

How are dependencies defined by requirements.txt different than dependencies defined by pyproject.toml or by setup.py?

If your answer is RTFM, maybe you can point me to some resources. I read a bit through Poetry documentation, but I still don't really understand those issues.

Re:Boring Python: dependency management (2022) by aragilar on Tuesday February 06, 2024 @02:34AM (#39271823)
The issue isn't poetry per se, but what simplifying assumptions (that there is a single solution and you can compute it on any system, and that "system effects" can be ignored/handled by wheels, see https://pypackaging-native.github.io/ which covers some of this) it tries to make to solve the solution of what to install.

"Build isolation" is all about limiting what is available at build time (this generally conflicts with inspecting the environment/system), and is vaguely (but no where near sufficient) for reproducible builds (see https://reproducible-builds.org/, which came from linux distros). Lockfiles are similarly an issue because they try to assume they can cover all cases (which they cannot, see e.g. various discussions about sdists for the lockfile PEP). Both these conflict with trying to work out what works for the current system (which is what pytorch tries to do because there are so many different options), or with trying to control what is discovered when trying to do the build on one system but run on another.

Re:Boring Python: dependency management (2022) by dagw on Monday February 05, 2024 @03:50AM (#39258806)
The fact that it only kind of sort of incidentally works some times shows clearly that people who consider PyTorch a vital library for their work aren't the target audience for poetry. Another use case they don't really care about are people for whom compiling C++ etc. is important part of their build process.

Poetry is great if you're developing the sort of software that the poetry devs priorities, but since poetry is only targeting a subset of the python community it will never be a good default tool for all python development, and thus the fragmentation continues.

Re:Boring Python: dependency management (2022) by chrisledet on Monday February 05, 2024 @02:15AM (#39258236)
Agreed, I haven't been too impressed with poetry.
Re:Boring Python: dependency management (2022) by Ultimatt on Monday February 05, 2024 @03:37AM (#39258691)
I'd say this is more like pytorch is horribly packaged and a nightmare as an eng user to integrate into codebases. All of their assumptions are you use conda and are a datascientist one time doing something in an ipython notebook. Very little consideration is made to how you deploy at scale reliably. It's not really on poetry.
Re:Boring Python: dependency management (2022) by dagw on Monday February 05, 2024 @03:57AM (#39258862)
It's not really on poetry.

While this is on the one hand at least partially true, it is also the case that as long a poetry can't deal with these (and similar) cases then it cannot really be said to be a candidate for the 'default' dependency management tool. Having people say 'just use poetry' as the go to answer is very bad default advice as long as poetry will fail in a number of important cases.

Re:Boring Python: dependency management (2022) by AzuraIsCool on Monday February 05, 2024 @09:58AM (#39261850)
But you can use poetry for it xd
Re:Boring Python: dependency management (2022) by hereonout2 on Monday February 05, 2024 @04:37AM (#39259166)
You can install pytorch with it, I have been doing so for a while.

  [tool.poetry.dependencies]
  torch = { version = "1.13.1+cu116", source = "pytorch-cuda" }
  
  [[tool.poetry.source]]
  name = "pytorch-cuda"
  url = "https://download.pytorch.org/whl/cu116"
  priority = "explicit"
But, it does seem to go off and download many GBs of pytorch packages for all possible python, architecture and OS versions which takes ages. Feels very broken at least.
Re:Boring Python: dependency management (2022) by andenacitelli on Sunday February 04, 2024 @10:08PM (#39256839)
Hard agree. This article makes good points but I feel like Poetry just simplifies so much of it and makes it so much less manual. We’ve used it at $DAYJOB for months and it’s been awesome. In the same sense as the article — it’s been sufficiently boring to rarely be the source of problems, and when it is, it’s almost always our fault rather than a design flaw in Poetry.
Re:Boring Python: dependency management (2022) by EdwardDiego on Sunday February 04, 2024 @09:35PM (#39256662)
Notably Poetry lockfiles are farrrrrr better cross-env then Pipenv's, which if you lock on an ARM Mac, will not install on a x64 Linux box. Poetry just works.
Re:Boring Python: dependency management (2022) by BiteCode_dev on Monday February 05, 2024 @01:44AM (#39258069)
I'm going to sound like a zealot at this point but:

'"Why not tell people to "simply" use pyenv, poetry or anaconda'

https://www.bitecode.dev/p/why-not-tell-people-to-simply-use

Re:Boring Python: dependency management (2022) by tempaccount1234 on Monday February 05, 2024 @03:41AM (#39258724)
Why not simply use pip? At least that works…
Re:Boring Python: dependency management (2022) by BiteCode_dev on Monday February 05, 2024 @04:42AM (#39259192)
Well, since I'm spamming, let's commit to it :)

That's pretty much what "Relieving your Python packaging pain" advocates.

https://www.bitecode.dev/p/relieving-your-python-packaging-p...

Re:Boring Python: dependency management (2022) by aragilar on Monday February 05, 2024 @07:52AM (#39260720)
I agree with your posts(s), but conda is different to pyenv, poetry and similar tools (rye, hatch etc.) in that unlike the others which only manage Python, it (tries) to manage your whole environment. Can the others manage R packages (because your using r2py or reticulate)? What about installing pandoc? Or latex?

I suspect we're going to continue in having churn in what people seem to be calling project/environment managers simply because there's a bunch of possible workflows (given the various possible deployment scenarios), and there's enough library support that starting your own is a weekend project (and then all the corner cases are discovered and you make a choice that works for some but not others and the cycle begins anew). conda seems to have dominated its niche (I haven't seen activepython nor enthought in years), but I suspect that part of the ecosystem is more naturally stable.

Re:Boring Python: dependency management (2022) by tlonny on Monday February 05, 2024 @03:34AM (#39258671)
Very tangential, but wanted to share a small QoL trick that makes using poetry (and venv in general) much nicer...

Instead of having to type "poetry shell" to activate the virtualenv, I use a tool called direnv that automatically modifies envars when you enter a directory - undoing the changes when you leave.

In my direnv config (.envrc) for the directory I write:

   PATH_add .venv/bin
To add all the virtualenv executables to PATH. Now I can just call "python/pip" directly and never have to worry if I'm inside my virtualenv or not.

N.B. for this you need to update poetry config to ensure virtualenvs are placed in the project directory vs. the default centralised location

Re:Boring Python: dependency management (2022) by PufPufPuf on Monday February 05, 2024 @04:06AM (#39258923)
It's better to activate the virtualenv properly:

. .venv/bin/activate

Setting up just the PATH may confuse some tools.

Re:Boring Python: dependency management (2022) by regularfry on Monday February 05, 2024 @07:11AM (#39260372)
I personally intensely dislike tools that change the environment based on the directory contents. I use an `exec` script in the root (or parent, sometimes) that just does this:

    ```
    #!/bin/bash

    source .venv/bin/activate
    exec $SHELL $@
    ```
That gives me a subshell with the virtual env active, and the path to the virtual env goes into $PS1 so I've got a very visible "you are working in this project" signifier.

I don't know why subshells aren't more popular for this sort of thing, they remove a lot of ambiguity (and subsequent opportunities to screw things up) for me.

Re:Boring Python: dependency management (2022) by mrweasel on Monday February 05, 2024 @03:45AM (#39258770)
Poetry solves a very real problem, but I kinda hate it. It's not a pleasant tool to use.

Allow me to out-bore you, because there's way to much excitement in your solution. We just build Debian packages and deploy our applications that way. The Debian packaging infrastructure could be better, but mostly it just wraps around setuptools/pyproject. You need to rebuild your containers every time there's a security update to package, we just set apt to auto-update.

Re:Boring Python: dependency management (2022) by alfons_foobar on Monday February 05, 2024 @04:19AM (#39259033)
We use poetry inside docker in a very similar way, but use a multistage docker file - the "builder" image has poetry installed and installs the application into a venv, we then copy that venv to the "prod" image and run the application there (so we do not have poetry or any other packages that are only required at build time in the final image).

Any thoughts on that approach?

Re:Boring Python: dependency management (2022) by linkdd on Monday February 05, 2024 @04:31AM (#39259116)
For Django projects, I made a gist as a template of what I use: https://gist.github.com/linkdd/4aac2c2efc4a51af6ca4b05f395de...

In there, you'll find the settings.py split by environment (like you would have in the Elixir world), and a multistage Docker image designed to be cache-friendly (for faster Docker builds). Some stages even run the test suite, so it could be compared to a CI pipeline.

Not having build dependencies in the production image is definitely a good approach.

Re:Boring Python: dependency management (2022) by irjustin on Monday February 05, 2024 @07:57PM (#39269396)
It's definitely fine with no problems. We used to do this too.

But, it optimizes for something that only matters if you're in a space constrained setup. Turns out for "almost-all" web setups, a few extra hundreds of MBs don't matter even a few GB possibly.

Even in AWS Lambda where cold-start is king, size doesn't matter[0] - I am the top answer. As long as you have fast starts and that's determined by code paths, libraries etc.

So as long as your program readies itself quickly, having gcc and a few other std libs don't impact any key metric.

[0] https://stackoverflow.com/questions/69512271/will-the-cold-s...

Re:Boring Python: dependency management (2022) by baggiponte on Monday February 05, 2024 @06:04AM (#39259800)
Not a huge fan of poetry since it does not comply with standards (PEP621 PEP518...). But it has a great UX compared to alternatives. My go-to has been PDM: which is basically PEP-compliant poetry. But it is being maintained by one dev only and support for containers is not great...
Re:Boring Python: dependency management (2022) by awongh on Monday February 05, 2024 @06:15AM (#39259913)
Also coming from Rails, besides the way pip, etc. works, I was also surprised at just how fragmented the landscape is. And with a lot of strong opinions on how that person’s viewpoint is correct. When I was figuring this stuff out it was very confusing because of course these strongly held opinions neglect to list the huge downsides. I think it’s an interesting note about the culture of Python.
Re:Boring Python: dependency management (2022) by irjustin on Monday February 05, 2024 @05:39PM (#39268221)
YES!! 100%, I emphatically CANNOT stand how terrible model validations are in Django (obvious vs Rails) and they are a direct result of "we do it better".

full_clean vs Form.full_clean vs DRF. It's just an absolute shit show.

Even if it is better, it's just annoying and not worth the benefits.

For us the overall benefits outweigh the drawbacks but I get angry every time when I need to do a model validation.

Re:Boring Python: dependency management (2022) by brnt on Monday February 05, 2024 @08:13AM (#39260887)
What does Poetry provide that the 'official' PyPA tooling doesn't? I'm referring to the somewhat unfortunately named 'wheel' and 'build'? Also, `pyproject.toml` itself is an innovation from outside of Poetry afaik.

And for what use-cases does Poetry not work, if any?

Re:Boring Python: dependency management (2022) by irjustin on Monday February 05, 2024 @05:35PM (#39268180)
... i mentioned it in my post.

but, it is fair to say, if you've never experienced a versioning issue then you wouldn't find poetry amazing.

Re:Boring Python: dependency management (2022) by brnt on Tuesday February 06, 2024 @07:38AM (#39273553)
How is it different from pip freeze -l > requirements.txt?
Re:Boring Python: dependency management (2022) by rirze on Tuesday February 06, 2024 @11:02AM (#39275867)
Poetry is a step up from plain pip but it's not complete either. I recommend switching to PDM. PDM follows more Python standards and is faster too.