Re:Machine Learning: The High Interest Credit Card of Techni...
by malux85 on Tuesday October 06, 2015 @04:11PM
(#10341975)
I agree - it's so challenging to keep up with Terabytes and Terabytes of moving data ... but reproducibility is so fundamental to science in general.
I'm not disagreeing, I'm just wondering if there's somehow a middle ground. What do you think? Can you share any experiences or tips?
I am running my own company at the moment, and my main dataset is about 2.5 TB, it's a very skinny table (it's actually a 191,000,000 x 6,000,000 matrix). It takes too long to iterate the whole dataset, so when I'm training my machine learning classifier I subsample the large dataset then compare previous model results to the new model, and inspect the differences.
Re:Machine Learning: The High Interest Credit Card of Techni...
by binarymax on Tuesday October 06, 2015 @04:55PM
(#10342315)
If I have one piece of advice to give (that you probably already know), it is to have a sound way to measure results and ensure that you are improving. Usually that involves custom tooling that falls within the integration test layer. So if 'inspecting the differences' is a reproducible and statistically reliable way to know that you are good - then you probably are!
Then all of a sudden you have terabytes of data updating a model faster than you can keep up with it, and after a couple months, starting from scratch to get the same model doesn't seem plausible anymore. Two years of production time and you have a honking mess.
Also it is easy to fall into the trap of prototype-to-production, when you've developed a nice looking demo that suddenly has all of the real customers and none of the engineering rigor.
It is an interesting time to be in software :)