The Hidden Cost of Using Laptops for Data Science

Re:The Hidden Cost of Using Laptops for Data Science by fphhotchips on Tuesday May 09, 2017 @09:30PM (#14305276)

Agree with the main thrust, but think that it misses a couple additional key costs: lack of reproducibility and lack of collaboration. Both of these are actually covered by the product that Domino sells, too!

Essentially, work done on laptops often doesn't make it past the laptop. Perhaps the modelling code makes it into Git, but the test data gets lost, the process that lead to it gets lost, etc. etc. Then reproducing and potentially modifying the work that has already been done becomes a labor and the work gets left behind or completely redone as a result. Let alone all the differences in environment and setup that can manifest between different laptops.

[ Parent | on HN ]

Re:The Hidden Cost of Using Laptops for Data Science by dwhitena on Tuesday May 09, 2017 @09:51PM (#14305360)

Reproducibility and provenance are essential for the data science workflows. And it's important to maintain these at scale. Take a look at Pachyderm some time as well if you get a chance (I work on the project for full disclosure). We version data in addition to code for full reproducibility. Even for distributed multi-stage and multi-language pipelines.

[ Parent | on HN ]