News for hackers. Stuff that matters.
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by macNchz on Wednesday February 10, 2021 @06:32PM (#26096301)
Poetry (and now pip with its new resolver) has taken a hard line on not letting you install conflicting dependencies(1)(2). On the surface this sounds ideal: you can't wind up with some combination of packages that won't work together.

The reality is that there are tons of Python libraries out there with poorly specified (or just slightly out of date) dependencies, and it's super annoying to have your package manager just give up and say "sorry, not installing that!" when you encounter them. This is an issue when working on applications with lots of dependencies, where I know certain packages will work together despite their setup.py files claiming otherwise.

There are many reasons you'd want to install a version of a library that disagrees with one of your other dependencies...security updates in subdependencies, maintainers who are slow to test against new versions or don't even use their own libraries anymore etc. The answer in these cases shouldn't be limited to "work with the package maintainer to get the dependencies changed, or fork it". Yarn, for example, has a simple way to handle this situation (and a clear understanding of why it's necessary baked into the docs!) (3)

[1] https://github.com/python-poetry/poetry/issues/697

[2] https://github.com/pypa/pip/issues/8076

[3] https://classic.yarnpkg.com/en/docs/selective-version-resolu...

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by rendaw on Wednesday February 10, 2021 @09:13PM (#26097470)
They also don't let you blacklist bad dependencies or force certain versions of transitive dependencies.

Poetry is much better than Pipenv and... text files with lists of packages to install. I introduced it at my workplace because of that, but I'm terrified that we'll at some point need some poorly maintained 3rd party dep and end up with a hack that was worse than anything we had before (like, a poetry wrapper that modifies the lock file after every operation).

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by MrPowers on Wednesday February 10, 2021 @10:23PM (#26097914)
Python workflows have gotten way better in the last few years:

* Poetry for dependency management

* pyenv for Python version

* Black for automated code formatting

The .python-version file automatically sets my Python version. I run black . to format the entire project. I can run poetry add XYZ to add a dependency to the project. poetry publish deploys my open source projects to PyPi.

I wrote this blog post that explains how to use Poetry + Pandas + Jupyter if you're interested in learning more: https://mungingdata.com/python/jupyter-workflow-poetry-panda...

Poetry-like programs have existed for a long time. I used Bundler with Ruby when I started programming in 2012 and was shocked with the state of Python dependency management when I was experimenting with the language a few years ago.

The ecosystem has made huge advances and now mypy is making it even better. It's great to see all these languages adopting all the amazing Scala language features!

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by woile on Thursday February 11, 2021 @04:13AM (#26099673)
I've been using the same setup, I'd add only `python -m venv .venv`

I find it way simpler than before, it was worth learning to use those tools.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by arusahni on Wednesday February 10, 2021 @03:58PM (#26094688)
I personally still find Poetry and Pipenv to be a little heavy-handed for my preferred workflows. To that end, pip-tools [1] is what I recommend to those who feel similarly. It balances pinning and dependency management with common Python practices (e.g., requirements.txt).

1: https://github.com/jazzband/pip-tools

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by epage on Wednesday February 10, 2021 @09:59PM (#26097767)
My problem with pip-tools is that it only resolves requirements.in files for your existing platform (OS + Python version). Say I'm testing something in CI for a bunch of platforms, that means to lock my dependencies I need all of those platforms locally. I run into this as well when distributing internal tools at my company, especially since we force requiring hashes.
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by globular-toast on Thursday February 11, 2021 @03:32AM (#26099464)
Sounds like you're doing it wrong. Pip-tools is for building environments. The common example is a docker container. In that case you control the environment completely and you use pip-tools to ensure the images you build are reproducible.

But most python packages are not shipped as environments. They are packages to be installed into user's environments. So you shouldn't be locking your environment when you test it, you should be installing it into a fresh environment like a user would do. You can test against specific major versions of your dependencies, but this is something handled well by tox and doesn't need any locking. In fact, locking will only make your life more difficult as you'll be targeting specific minor versions.

Long story short, use pip-tools if you're building docker images or otherwise deploying a tested environment. Don't use it for regular Python packages.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by scrollaway on Wednesday February 10, 2021 @04:19PM (#26094966)
The only reason to use pip-tools is because you already know it and you don't want to switch (yet). pip-tools is squarely worse than using Poetry.

Poetry is not perfect, mind you, but it's the absolute best we have in the Python world. Whenever I jump into a new project, switching it to Poetry is one of the first things I do so I have a sane working environment for it.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by arusahni on Wednesday February 10, 2021 @04:32PM (#26095109)
I'm glad tools like Poetry exist, as they're filling pretty horrible gaps in the Python ecosystem.

> The only reason to use pip-tools is because you already know it and you don't want to switch (yet).

Actually, I settled on pip-tools _after_ trying Poetry :-)

> pip-tools is squarely worse than using Poetry.

News to me. Like I said in my original comment: "I personally still find Poetry and Pipenv to be a little heavy-handed for my preferred workflows." A tool that gets out of my way and lets me achieve my goals is a better tool for me. Since Poetry can't meet me where I am, pip-tools is better _for me_.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by coffeefirst on Wednesday February 10, 2021 @06:04PM (#26096058)
I've actually backed away from Poetry after some really bad experiences.

When it works, it's damn near perfect.

When it doesn't work, and that usually involves a problem with a downstream dependency that isn't technically Poetry's fault, it's a massive hassle and sometimes impossible to manually override.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by kyllo on Thursday February 11, 2021 @01:51PM (#26105561)
I went from Pipenv to Poetry and then back to Pipenv after encountering an infinite loop in Poetry's solver on multiple separate occasions.
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by diragon on Thursday February 11, 2021 @02:50AM (#26099250)
I've submitted pull requests to projects when that happens. Not many yet, but all of them have been accepted in less than a week's time. People should do this and then the problem would eventually vanish.
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by ksm1717 on Wednesday February 10, 2021 @05:06PM (#26095440)
These are bold and absolute claims in response to proposing that an alternative may be better for a given user’s preferences.

“I prefer pepsi because coke is too sugary.”

“The only reason you should drink Pepsi is if you already have some. It’s a squarely worse drink than coke.”

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by jrib on Wednesday February 10, 2021 @06:02PM (#26096039)
pip-tools has the advantage that it is super-simple to understand and reason about.

poetry works well enough but it seems to bring more complexity with it.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by neolog on Wednesday February 10, 2021 @07:47PM (#26096858)
The complexity is the same or less with Poetry. It's just about whether you're delegating the complexity to Poetry or managing it yourself.
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by joshbuddy on Wednesday February 10, 2021 @04:29PM (#26095072)
How is it strictly worse?
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by AceJohnny2 on Wednesday February 10, 2021 @08:50PM (#26097301)
At this point, I'd like a block diagram of Python's various package manager tools, to show me which ones rely on what other ones, and which sit alongside.

Poetry uses Pip, right? Where does setuptools fit? distutils? What are wheels, eggs?... Where does Poetry fit vis-a-vis virtualenv?...

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by porkloin on Wednesday February 10, 2021 @09:06PM (#26097413)
Yeah, it's bananas.

I started a new job last year at a place that has a large percentage of their codebase in Python after many years of working with OO and relatively "modern" PHP. Of course, taking PHP's reputation online to heart, you can imagine my glee! To know that I would soon be working with a language that has a little more respect, clout, and professionalism!

Oh, how wrong I was.

One of the first things I noticed was the package management disaster, along with the virtual environment and local/global package issues. Just shy of a year later, and I'm absolutely flabbergasted that Python doesn't get utterly roasted online for how bad and fragmented some of the tooling is. I could go on about other frustrations I have with the broader Python ecosystem, but I'll just leave it at the point that Python folks shouldn't cast _too_ many stones in the direction of JS or PHP without knowing what the current state of tooling in those languages are.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by twunde on Wednesday February 10, 2021 @11:28PM (#26098264)
As a former PHPer myself, I agree with you on all counts. Python's tooling, especially around package management is _terrible_, hand's down the worst of any major dynamic language, even comparing where they were 5 years ago to where Python is today. Virtual environments are Python's "solution" for local installs, where everyone else supports local installs by default. There's no standard way to specify dev dependencies, they're just all dependencies. Thankfully now most things are pip installable without problems unlike 5-10 years ago, but you'll still occasionally run into weird easy_install/setup/wheel issues.

I'll note that my other complaint is the quality of many open source libraries are a significant step below their counterparts in PHP, Ruby or JS. Django, boto and presumably the data science libraries are top notch, but after that even heavily used projects like celery lack the level of polish that major projects in other languages have.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by Isinlor on Thursday February 11, 2021 @04:53AM (#26099851)
My experience too. I was shocked how terrible Python ecosystem is.

For example stuff like dependency confusion i.e. public package with potentially malicious code overwriting private package are impossible in Composer by design.

https://twitter.com/seldaek/status/1359417655762055171

But pip, npm and RubyGems failed to protect against dependency confusion. So Apple, Microsoft and dozens of other companies pulled malicious code.

https://medium.com/@alex.birsan/dependency-confusion-4a5d60f...

We are very lucky to have awesome Jordi Boggiano in the PHP ecosystem.

He is doing gods work on Composer and Packgist.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by Doxin on Thursday February 11, 2021 @03:18AM (#26099398)
As a long-time python dev: ignore 90% of the whole packaging situation. Use a virtualenv for each project, use pip with a requirements.txt to install packages.

The above works fine. I honestly don't know what poetry adds aside from a different way to invoke the same tools.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by Isinlor on Thursday February 11, 2021 @04:57AM (#26099870)
How do you lock all dependencies to a hash?

Let's says if one of packages that you depend on decided to depend on some third package. The maintainer of that third package decides to stop maintaining it and gives it to someone else. This someone else decides to inject bitcoin mining into the third package.

How will you know with pip that you are not pulling malicious code into production?

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by disgruntledphd2 on Thursday February 11, 2021 @06:49AM (#26100529)
How do you handle C/C++ dependencies?

This is super common in the DS/ML space, and until recently, pip was actively unhelpful (i see you have numpy version x, let's install version y silently because you definitely hand review all the transitive dependencies before installing a new package).

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by Isinlor on Thursday February 11, 2021 @07:14AM (#26100664)
In that case building a container image and testing it before deployment seems to be the easiest solution, certainly not ideal.

Otherwise, I have no idea how to handle cross language dependencies well. My guess would be that operating system package managers ought to handle it, but I never tried to work with something like Debian repositories. Judging by what I've heard from other people it's a lot of pain.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by Doxin on Thursday February 11, 2021 @07:15AM (#26100669)
These days ideally those dependencies would ship a wheel which solves the problem.

If they don't you are expected to have a working C/C++ compiler set up which is definitely not ideal. At least on linux it's often as simple as installing a build-essentials package. No clue what that'd be like on other OSes since I don't use them.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by disgruntledphd2 on Friday February 12, 2021 @04:40PM (#26118881)
Only if the wheels aren't incompatible.

I think my issues are really with pip and it's dependency resolver. Give me conda every day, it may take a while but it handles compiled dependencies and doesn't break my environment without at least telling me.

And yes, I use docker containers now.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by Doxin on Thursday February 11, 2021 @06:57AM (#26100565)
Well presumably under that threat model you'd run your own package cache. But in practice that doesn't really seem to be a thing that happens. Python hasn't had the equivalent of left-pad yet as far as I know.
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by Isinlor on Thursday February 11, 2021 @07:03AM (#26100608)
You may want to take a look: https://medium.com/@alex.birsan/dependency-confusion-4a5d60f...

Security by "it hasn't happen yet" is not a security.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by Isinlor on Wednesday February 10, 2021 @04:07PM (#26094803)
One thing I fail to understand is why poetry is installing everything all over the system instead of keeping code contained to the project directory. If I delete my project I want get rid of everything related instead tracking down where what was installed.

But that seems to be an issue with every other tool in Python too.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by scrollaway on Wednesday February 10, 2021 @04:17PM (#26094935)
You can set virtualenvs.in-project to use a local .venv folder instead (like you would have in the JS world). I use that a lot in CI and Docker images.

https://python-poetry.org/docs/configuration/#virtualenvsin-...

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by gooseyard on Wednesday February 10, 2021 @06:40PM (#26096356)
I've spent the last couple of weeks experimenting with converting some work-related modules to use Poetry, and although I immediately loved a few of the improvements, there are a few drawbacks that weren't immediately obvious.

Chiefly, the inability of the combination of poetry and pip to create editable installs of a module is a drag. Using "poetry shell" is ok, but many of the users of the module I maintain are accustomed to using pipenv or plain old virtualenvs and I'm as loathe to try to induce them to use some other tool for this as I would be if the tables were turned.

The other more serious issue is that the modules I take care of are internal things in a private PyPI repository, and poetry unfortunately has a few bugs in this area. Working fixes are in PRs that have been sitting for months, and I've attempted to contact what I think is the project maintainer without any response. In my opinion, that's absolutely fine; this is a volunteer effort, I certainly don't expect the volunteers to do heroics to support me, and I can come up with some way of getting the patches that I need into the tool.

But at the same time, the ideal tool for dealing with Python packaging is going to need to be able to coordinate this sort of stuff with the pip maintainers, and to have some mechanism to support corporate users and all that boring stuff. I want to be enthusiastic about this one because it definitely seems to knock off a lot of sharp corners in python packaging, but I'm not sure if its got legs.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by the_jeremy on Wednesday February 10, 2021 @08:10PM (#26097017)
Exactly my problem. Poetry 1.1.0 through 1.1.4 (aka every release since Jul 2020) are broken on our private PyPI instance. If you maintain OSS python packages, poetry might be great for you, but it's apparent the maintainers aren't willing to support enterprise use, and I wish that was made clear to potential users.
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by neolog on Wednesday February 10, 2021 @06:46PM (#26096409)
`poetry install` installs the current project in editable mode.
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by gooseyard on Wednesday February 10, 2021 @07:55PM (#26096912)
it does, but most of the users of this module are accustomed to being able to run "pip install -e ." in order to get the module editable in their virtualenv, and since this module has dependencies that are also in the private repo, the issues that I alluded to (which affect private repos using artifactory which require client certs) cause trouble.

I'm still optimistic that these things will get fixed, but it's not going as smoothly as I'd hoped.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by Caligatio on Wednesday February 10, 2021 @11:06PM (#26098150)
I actually wrote the client cert code back in the day, what problems are you having?
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by gooseyard on Thursday February 11, 2021 @09:16AM (#26101691)
When a cert is specified for a repo, the solver will use the cert and ca list when querying for packages, but after solving won't pass the cert and verify values when constructing the requests session to download the packages. There's a fix in https://github.com/python-poetry/poetry/pull/3490, but it has been pending review for several months.

There's a second issue (this one isn't cert related) where repos that return a 403 when a given package doesn't exist causes the solver to abort, which is problematic if your private repo is Artifactory, since the solver interprets this as an auth misconfiguration and won't fall back to the primary repo.

Based on some comments in the threads for those PRs I wondered if maybe the delay in merging was that some changes to the way repositories are configured, but it'd be great to have a temporary workaround for the time being.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by koolk3ychain on Wednesday February 10, 2021 @05:33PM (#26095723)
I still use pipenv and it's great. Python version management is a hell hole when sharing a project with other devs, pipenv solves this issue and is easy enough to containerize.
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by MrPowers on Wednesday February 10, 2021 @10:30PM (#26097949)
There were some pipenv red flags back in 2020, see the bottom portion of this blog post: https://jacobian.org/2019/nov/11/python-environment-2020/

Just checked the commit history of the repo and looks like pipenv is back on track. Do you have any additional context?

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by ehutch79 on Wednesday February 10, 2021 @03:46PM (#26094530)
I'm not sure why pip installing the dependency at the pinned version is an issue. Isn't that the point?

I feel like a lot of the issues people put forth as a reason for poetry and pip alternatives are a bit contrived, and more this is how npm|yarn|whatever do it expectations.

Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by asplake on Wednesday February 10, 2021 @03:50PM (#26094577)
Because it’s now down to you to manage which versions of which packages go with what versions of what other packages. When you really need to upgrade one and it fails for version compatibility reasons, it can be a genuinely difficult problem.
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by memco on Wednesday February 10, 2021 @05:49PM (#26095900)
Does anyone here have recommendations for using poetry or a similar tool for nested projects? We have a suite of separate repos that all have separate requirements but are really sub-repos of the larger project. It would be nice to have a way to poetry install and manage them for development from a single place while still being able to install the separate components when we deploy.
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by kbd on Wednesday February 10, 2021 @04:58PM (#26095360)
Poetry has been great, for both local development and for deployment. I recommend using direnv (which has a Poetry recipe) to automatically activate your Poetry virtualenvs locally.
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by jdfellow on Wednesday February 10, 2021 @10:26PM (#26097935)
The direnv layout for poetry works wonders!
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by pvillano on Thursday February 11, 2021 @10:04AM (#26102152)
Or you can keep an un-versioned* requirements.txt, and every month/CVE re-freeze it

  #!/bin/bash
  docker run -v "$(pwd)":/req -w /req python:3.9.1-slim-buster \
  sh -c "pip install -r requirements.txt \
  && echo \"#$(date)\" > requirements-frozen.txt \
  && pip freeze >> requirements-frozen.txt"
* as pinned as you like: major, minor, greater than etc.
Re:A Poetic Apology: Or Why Should You Use Poetry to Manage ... by grawp on Wednesday February 10, 2021 @05:48PM (#26095880)
Poetry is for Python kinda what Cargo is for Rust.