News for hackers. Stuff that matters.
Re:Processing 40TB of code from 10M projects with a dedicate... by twic on Tuesday October 01, 2019 @09:26AM (#21125048)
I am an utter noob at Python packaging, but i would like to learn more, so excuse the basic question, but: why doesn't it work? And is there anything in the ecosystem which is like that, but does work?

I had heard that the Python packaging and deployment story had got a lot better in recent years, but this sounds like it still falls far short of table stakes.

Re:Processing 40TB of code from 10M projects with a dedicate... by weberc2 on Tuesday October 01, 2019 @09:43AM (#21125225)
I'm not sure why the Python community can't get it sorted out. We've been using Pipenv because it's one of a few tools that supports lockfiles/reproducible builds, and at the time we chose it, the Python packaging authority was advertising it as the official solution for Python packaging; however, after we were already invested in it, they backpedaled becaused the community realized it was very buggy and super slow (as in "it takes literally half an hour to add or remove or update any dependency").

We're finally getting around to migrating away, and we've settled on Twitter's github.com/pantsbuild/pants which is like Google's Bazel except that it's jankier in every way _except_ that Pants supports Python while Bazel doesn't (Bazel advertises Python support, but it doesn't work as the many Python 3 issues in their issue tracker can attest). The nicest thing about Pants is that it builds pex files (github.com/pantsbuild/pex) which are executable zip files that ship all of your dependencies except the Python interpreter and shared objects.

I'm still not very satisfied with this solution, and it's still far, far worse than Go's dependency management and packaging model, but it's a dramatic improvement over Pipenv, virtualenv, etc (gives us both reproducibility and performance).

Re:Processing 40TB of code from 10M projects with a dedicate... by steeleduncan on Tuesday October 01, 2019 @10:50AM (#21125903)
Its not that Python packaging doesn't work, or that it is impossible to automate, but whatever you do it requires nontrivial time and effort to get right.

The point is that the static binary created by go is a single file that you copy into place and run. It is a lot simpler.