News for hackers. Stuff that matters.
Re:Docker for data scientists: Introduction and use cases by iamwil on Thursday January 04, 2018 @03:39PM (#16073863)
How did you manage the dependency versioning problem? Did you set up automated tests for that? Or did you just make sure everyone installed the same version with a requirements.txt file?
Re:Docker for data scientists: Introduction and use cases by SatvikBeri on Thursday January 04, 2018 @05:39PM (#16074779)
Anaconda has a better tool for cloning identical environments as long as both machines are on the same OS: https://conda.io/docs/user-guide/tasks/manage-environments.h...

In theory, pip lets you do this with pip freeze, but I found that pip often broke on some finicky libraries, while Anaconda failed much less frequently (though still has occasional issues.) The main drawback is the same OS requirement – this makes it harder to e.g. develop on a mac and then launch a pipeline on an Ubuntu machine in AWS.

Re:Docker for data scientists: Introduction and use cases by pwang on Thursday January 04, 2018 @06:03PM (#16074941)
What do you mean by "same OS requirement"? Can't you take your environment.yml from a Mac and recreate the environment on Linux?
Re:Docker for data scientists: Introduction and use cases by SatvikBeri on Thursday January 04, 2018 @06:37PM (#16075129)
Anaconda has two environment-copying tools, one is environment.yml (which is analogous to pip's requirements.txt), and the other is the "explicit" spec file, which actually points to specific files for specific packages. I've found environment.yml to fail with finicky libraries, same as requirements.txt, while the explicit spec file almost always works.