Re:Processing 40TB of code from 10M projects with a dedicate...
by folkrav on Tuesday October 01, 2019 @09:22AM
(#21125012)
Why is it ridiculous? You'll need the related libraries either case, and by default virtualenv uses symlinks. The "full virtualenv" part makes it sound like it's heavy, or something like that.
Re:Processing 40TB of code from 10M projects with a dedicate...
by theshrike79 on Tuesday October 01, 2019 @03:12PM
(#21129390)
If you do it in a language that provides static binaries or an equivalent, you don't need to pull dozens of libraries with possibly hundreds of files just to run a single script.
Imagine a situation where you'd need to perform an operation on a hundred servers, would you either transfer one static exacutable or build an virtualenv in every machine and download correct versions all of the related libraries?
Re:Processing 40TB of code from 10M projects with a dedicate...
by folkrav on Tuesday October 01, 2019 @04:52PM
(#21130488)
If I controlled the environment, I'd probably build a Docker image and deploy that. Otherwise yes, This kind of extreme example isn't very representative of the initial situation though. And you're also talking about switching up languages, which isn't always possible or easy, especially in large teams or codebases.
Re:Processing 40TB of code from 10M projects with a dedicate...
by mtberatwork on Tuesday October 01, 2019 @07:05AM
(#21124059)
> The server might have the wrong Python version available
That's the point of using virtual environments, so that you can run the Python version and libraries that you need. Also, as of 3.3, Python ships with venv which means you don't need to separately install virtualenv anymore. It's all very portable.
Re:Processing 40TB of code from 10M projects with a dedicate...
by tseabra on Tuesday October 01, 2019 @08:37AM
(#21124637)
You'd need to install the right Python version though, and even with something like pyenv you might need to install system packages.
It really isn't as portable as one might wish.
Re:Processing 40TB of code from 10M projects with a dedicate...
by penagwin on Tuesday October 01, 2019 @10:34AM
(#21125733)
This is my exact situation right now. Our website is made in Django, and we use Pyenv to manage our python versions. We'll likely be going to route of a docker image/ pyenv install script. It takes trial/error to figure out some of the packages you need if they aren't installed already, a single binary would be far easier for sure.
Re:Processing 40TB of code from 10M projects with a dedicate...
by oblio on Tuesday October 01, 2019 @12:56PM
(#21127669)
Portable means something else. Portable would mean that I could ship the virtualenv. Python missed the forest for the trees.
Re:Processing 40TB of code from 10M projects with a dedicate...
by theshrike79 on Tuesday October 01, 2019 @03:15PM
(#21129426)
I still need to have python installed on the machine and download all of the libraries needed.
Not all operating systems carry Python >3.3 by default, that would mean installing backports or unofficial repositories on production machines.
Or you can use a static binary that just works.
The server might have the wrong Python version available (2.x vs 3.x, multiple evolutions of 3.x with big features added in each point release). Et cetera, et cetera.
Or you could just do it with bash or a static Go binary and be done with it. Portable, works pretty much everywhere.