So basically the single most exciting thing to ever happen to the ecosystem isn’t supported by Poetry.
Not sure where I would stand if I fully investigated it tho.
[0] https://github.com/python-poetry/poetry/issues/6409#issuecom...
Due to Poetry's architecture, it can't satisfy all three at the same time:
- the platonic ideal of build isolation and lockfiles
- installing the appropriate accelerator specific version of pytorch for the current platform non-interactively or one that the user selects
- dependencies for pytorch that are transitively compatible with the way dependencies for pytorch are expressed elsewhere
This is something you can achieve with setuptools and setup.py by forfeiting the platonic ideals of build isolation and lockfiles.
Poetry, on the other hand, does not let you choose which lamb to sacrifice. Everyone in the thread, for the last two years, who has reported that they have had some success are misunderstanding the state of their install, and have interacted with flaws in all three situations I'm describing. Usually they have something that will not correctly install anything that is dependent itself on PyTorch, which is essentially useless, since everything in the PyTorch ecosystem is is dependent on it, and the main workaround the community uses - installing torch first, followed by installing dependencies from a requirements.txt, followed by copying a dump of scripts - is not compatible with poetry.
Here's the facts of the matter:
- 1.2M requirements.txts https://github.com/search?q=path%3A%2F%5Erequirements.txt%24...
- 664k setup.pys https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F&type...
- setup.pys that reference requirements.txts https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F+requ... 67.1k
- only 30k pyproject.toml specified with dependencies https://github.com/search?q=path%3A%2F%5Epyproject.toml%24%2...
Basically 2/3rds of Python end users do not engage with packaging at all. pyproject.toml with dependencies is about 1.5% of the ecosystem. It basically provides only downsides compared to setup.py and pinning your package versions by commit in your setup.py dependencies aka doing exactly what golang does, and this does not require any external tools in Python. In my opinion, the Poetry developers need to fix pytorch or they will not get adoption during Peak Python.
Frankly setup.py is a big hack and we should strive to do better.
What does it mean, build isolation? And how does PyTorch violate that? Or why can't we install PyTorch with build isolation?
What is the issue with lockfiles?
I can understand there might be issues if you need to install a custom accelerator-specific version of PyTorch (although also I don't fully understand what the issues are about this). However, what's with just the standard version, which is suitable for most people, as it comes with CUDA support?
What do you mean by non-interactively? pip install is non-interactively, or not?
What issues are there with the dependencies for PyTorch? Why are they not compatible with the dependencies expressed elsewhere?
How are dependencies defined by requirements.txt different than dependencies defined by pyproject.toml or by setup.py?
If your answer is RTFM, maybe you can point me to some resources. I read a bit through Poetry documentation, but I still don't really understand those issues.
"Build isolation" is all about limiting what is available at build time (this generally conflicts with inspecting the environment/system), and is vaguely (but no where near sufficient) for reproducible builds (see https://reproducible-builds.org/, which came from linux distros). Lockfiles are similarly an issue because they try to assume they can cover all cases (which they cannot, see e.g. various discussions about sdists for the lockfile PEP). Both these conflict with trying to work out what works for the current system (which is what pytorch tries to do because there are so many different options), or with trying to control what is discovered when trying to do the build on one system but run on another.
Poetry is great if you're developing the sort of software that the poetry devs priorities, but since poetry is only targeting a subset of the python community it will never be a good default tool for all python development, and thus the fragmentation continues.
While this is on the one hand at least partially true, it is also the case that as long a poetry can't deal with these (and similar) cases then it cannot really be said to be a candidate for the 'default' dependency management tool. Having people say 'just use poetry' as the go to answer is very bad default advice as long as poetry will fail in a number of important cases.
[tool.poetry.dependencies]
torch = { version = "1.13.1+cu116", source = "pytorch-cuda" }
[[tool.poetry.source]]
name = "pytorch-cuda"
url = "https://download.pytorch.org/whl/cu116"
priority = "explicit"
But, it does seem to go off and download many GBs of pytorch packages for all possible python, architecture and OS versions which takes ages. Feels very broken at least.'"Why not tell people to "simply" use pyenv, poetry or anaconda'
https://www.bitecode.dev/p/why-not-tell-people-to-simply-use
That's pretty much what "Relieving your Python packaging pain" advocates.
https://www.bitecode.dev/p/relieving-your-python-packaging-p...
I suspect we're going to continue in having churn in what people seem to be calling project/environment managers simply because there's a bunch of possible workflows (given the various possible deployment scenarios), and there's enough library support that starting your own is a weekend project (and then all the corner cases are discovered and you make a choice that works for some but not others and the cycle begins anew). conda seems to have dominated its niche (I haven't seen activepython nor enthought in years), but I suspect that part of the ecosystem is more naturally stable.
Instead of having to type "poetry shell" to activate the virtualenv, I use a tool called direnv that automatically modifies envars when you enter a directory - undoing the changes when you leave.
In my direnv config (.envrc) for the directory I write:
PATH_add .venv/bin
To add all the virtualenv executables to PATH. Now I can just call "python/pip" directly and never have to worry if I'm inside my virtualenv or not.N.B. for this you need to update poetry config to ensure virtualenvs are placed in the project directory vs. the default centralised location
. .venv/bin/activate
Setting up just the PATH may confuse some tools.
```
#!/bin/bash
source .venv/bin/activate
exec $SHELL $@
```
That gives me a subshell with the virtual env active, and the path to the virtual env goes into $PS1 so I've got a very visible "you are working in this project" signifier.I don't know why subshells aren't more popular for this sort of thing, they remove a lot of ambiguity (and subsequent opportunities to screw things up) for me.
Allow me to out-bore you, because there's way to much excitement in your solution. We just build Debian packages and deploy our applications that way. The Debian packaging infrastructure could be better, but mostly it just wraps around setuptools/pyproject. You need to rebuild your containers every time there's a security update to package, we just set apt to auto-update.
Any thoughts on that approach?
In there, you'll find the settings.py split by environment (like you would have in the Elixir world), and a multistage Docker image designed to be cache-friendly (for faster Docker builds). Some stages even run the test suite, so it could be compared to a CI pipeline.
Not having build dependencies in the production image is definitely a good approach.
But, it optimizes for something that only matters if you're in a space constrained setup. Turns out for "almost-all" web setups, a few extra hundreds of MBs don't matter even a few GB possibly.
Even in AWS Lambda where cold-start is king, size doesn't matter[0] - I am the top answer. As long as you have fast starts and that's determined by code paths, libraries etc.
So as long as your program readies itself quickly, having gcc and a few other std libs don't impact any key metric.
[0] https://stackoverflow.com/questions/69512271/will-the-cold-s...
full_clean vs Form.full_clean vs DRF. It's just an absolute shit show.
Even if it is better, it's just annoying and not worth the benefits.
For us the overall benefits outweigh the drawbacks but I get angry every time when I need to do a model validation.
And for what use-cases does Poetry not work, if any?
but, it is fair to say, if you've never experienced a versioning issue then you wouldn't find poetry amazing.
With the lock file being default, we don't worry about different installs in different envs.
Having come from the Rails world, Bundler system was solved for... a decade? So I was surprised it was such a mess in Python until so recently.
At the core, the thing that makes Poetry and Bundler system so predictable is 1, lock file. 2, ability to install different versions in diff locations and referencing the version you need to load. Each alone isn't enough.
npm had the same problem pip suffered from, you may have a version installed different what the req.txt, proj.js or even lockfile says but because it exists inside the location, it gets loaded. It wasn't until yarn2 did node_modules finally get moved out such that side-by-side versions wasn't awkward.
[EDIT]
If you're not using Poetry + Docker for deployment yet, I 100% recommend it as the "boring" method.
[0] https://packaging.python.org/en/latest/tutorials/packaging-p...