Re:Data science tips and tricks from the developer community
by gilbetron on Tuesday September 05, 2017 @10:43AM
(#15175666)
Is there a good solution to that problem, though? (Serious question). I recently did a laptop refresh and am using it as an opportunity to solidify my approach to ML development, and would love to hear if there is a good solution to long-term reproducibility. I'm currently leaning towards Docker, but maybe Vagrant or another "pure" VM approach is better...
[ Parent
| on HN ]
Re:Data science tips and tricks from the developer community
by KirinDave on Tuesday September 05, 2017 @11:12AM
(#15175983)
Not perfectly, but a good start is to keep all the software assets AND data assets you used to train the model.
There needs to be an immutable, high performance read data store that has a 30+ year plan for survival if we're really going to retool our world around expert systems.
Re:Data science tips and tricks from the developer community
by sosuke on Tuesday September 05, 2017 @11:18AM
(#15176046)
I think the only real problem times you'll find are when the architectures are changed. x86, arm, you'd probably want to port your solution images then if ever. There will always be folks emulating hardware in software on new architectures.
Re:Data science tips and tricks from the developer community
by tnecniv on Tuesday September 05, 2017 @12:40PM
(#15176834)
Well you can always fire up LaTeX and write a report. If detailed enough, that in conjunction with the data set should be enough to survive anything.
Re:Data science tips and tricks from the developer community
by delazeur on Tuesday September 05, 2017 @10:58AM
(#15175837)
Carve it onto stone tablets.
Re:Data science tips and tricks from the developer community
by KirinDave on Tuesday September 05, 2017 @11:30AM
(#15176191)
You joke it, but it's a major problem that our tech for very stable WORM media has lagged behind demand.
Our use of data has grown so much faster than our network capacity (and indeed, it seems like we're going to hit a series of physical laws and practical engineering constraints here). "Data has gravity" but the only way to "sustainable" hold a non-trivial volume of data for 20 years right now is to run a data center with a big dht that detects faults and replicates data.