News for hackers. Stuff that matters.
Re:Docker for data scientists: Introduction and use cases by wyattjoh on Thursday January 04, 2018 @01:00PM (#16072439)
I'd think that Pachyderm [0] would be mentioned in the article, but it wasn't. It uses Docker under the hood for the data pipelines. I've always wanted an excuse to do some data science...

[0]: https://www.pachyderm.io/

Re:Docker for data scientists: Introduction and use cases by jdoliner on Thursday January 04, 2018 @02:14PM (#16073156)
Thanks for the plug wyattjoh, I'm one of the founders of Pachyderm so I'd like to expand on what you said just a bit to clarify.

Pachyderm does use Docker under the hood, but we don't obfuscate it away, so Data Scientists get the full power of Docker (and most of the power of Kubernetes) at their fingertips. This means you can easily grab prefabricated environments such as Jupyter or Tensorflow container images and deploy them directly. Docker is so good for packaging environments we didn't want to conceal it.

On the other hand, we felt the data orchestration capabilities of Docker were pretty lacking for Data Science use cases, so that's where we've focused our energy. Volumes are a good basic tool for getting data to your code, but they're a pretty blunt instrument. How do you split data up to parallelize over it? How do you make sure you've got the right version of data? How do you schedule new computations when data becomes available? Those are some of the use cases we solve with our distributed file system PFS.

Re:Docker for data scientists: Introduction and use cases by haraball on Thursday January 04, 2018 @01:59PM (#16073013)
Singularity [0] is a related alternative.

[0]: http://singularity.lbl.gov/

Re:Docker for data scientists: Introduction and use cases by claytonjy on Thursday January 04, 2018 @03:03PM (#16073550)
+1 for pachyderm; my team and I are using it to build backtesting platforms for financial modeling, and it's been really nice to use. Also great in combination with Gitlab's built-in CI/CD stuff. Product aside, we've appreciated the team's availability on both Slack and Github too.
Re:Docker for data scientists: Introduction and use cases by raab on Thursday January 04, 2018 @01:04PM (#16072487)
Cool! I had never heard of that. Starred...