News for hackers. Stuff that matters.
Re:Processing 40TB of code from 10M projects with a dedicate... by skocznymroczny on Tuesday October 01, 2019 @08:53AM (#21124770)
virtualenv still requires internet access. You'd have to create a venv on destination machine and pip install your packages in. You can't scp a local venv onto destination machine and have it work.
Re:Processing 40TB of code from 10M projects with a dedicate... by AdamJacobMuller on Tuesday October 01, 2019 @11:41AM (#21126622)
I've done things in a way that it doesn't.

Basically build a virtualenv and then pack that into a .deb, install the .deb on the target machine and you have a self-contained package that requires no external resources to install.

That said, using golang is WAAAAY simpler. The virtualenv/deb solution, while it works, is very sketch.

Re:Processing 40TB of code from 10M projects with a dedicate... by turtlebits on Tuesday October 01, 2019 @11:36AM (#21126554)
You just need a venv with the right architecture, which is fairly trivial. You can then copy the venv directly over.

See this repo: https://github.com/unixtreme/D3Edit. It has Linux/Mac/Windows virtualenvs and it works without any additional setup

Re:Processing 40TB of code from 10M projects with a dedicate... by sswezey on Tuesday October 01, 2019 @12:46PM (#21127565)
I think what people are arguing is deploying a single binary is even more trivial than a "venv with the right architecture". Deploying one file is going to be easier than deploying a set of files.
Re:Processing 40TB of code from 10M projects with a dedicate... by twic on Tuesday October 01, 2019 @09:26AM (#21125048)
I am an utter noob at Python packaging, but i would like to learn more, so excuse the basic question, but: why doesn't it work? And is there anything in the ecosystem which is like that, but does work?

I had heard that the Python packaging and deployment story had got a lot better in recent years, but this sounds like it still falls far short of table stakes.

Re:Processing 40TB of code from 10M projects with a dedicate... by weberc2 on Tuesday October 01, 2019 @09:43AM (#21125225)
I'm not sure why the Python community can't get it sorted out. We've been using Pipenv because it's one of a few tools that supports lockfiles/reproducible builds, and at the time we chose it, the Python packaging authority was advertising it as the official solution for Python packaging; however, after we were already invested in it, they backpedaled becaused the community realized it was very buggy and super slow (as in "it takes literally half an hour to add or remove or update any dependency").

We're finally getting around to migrating away, and we've settled on Twitter's github.com/pantsbuild/pants which is like Google's Bazel except that it's jankier in every way _except_ that Pants supports Python while Bazel doesn't (Bazel advertises Python support, but it doesn't work as the many Python 3 issues in their issue tracker can attest). The nicest thing about Pants is that it builds pex files (github.com/pantsbuild/pex) which are executable zip files that ship all of your dependencies except the Python interpreter and shared objects.

I'm still not very satisfied with this solution, and it's still far, far worse than Go's dependency management and packaging model, but it's a dramatic improvement over Pipenv, virtualenv, etc (gives us both reproducibility and performance).

Re:Processing 40TB of code from 10M projects with a dedicate... by steeleduncan on Tuesday October 01, 2019 @10:50AM (#21125903)
Its not that Python packaging doesn't work, or that it is impossible to automate, but whatever you do it requires nontrivial time and effort to get right.

The point is that the static binary created by go is a single file that you copy into place and run. It is a lot simpler.