News for hackers. Stuff that matters.
Re:Machine Learning: The High Interest Credit Card of Techni... by malux85 on Tuesday October 06, 2015 @04:11PM (#10341975)
I agree - it's so challenging to keep up with Terabytes and Terabytes of moving data ... but reproducibility is so fundamental to science in general.

I'm not disagreeing, I'm just wondering if there's somehow a middle ground. What do you think? Can you share any experiences or tips?

I am running my own company at the moment, and my main dataset is about 2.5 TB, it's a very skinny table (it's actually a 191,000,000 x 6,000,000 matrix). It takes too long to iterate the whole dataset, so when I'm training my machine learning classifier I subsample the large dataset then compare previous model results to the new model, and inspect the differences.

Re:Machine Learning: The High Interest Credit Card of Techni... by binarymax on Tuesday October 06, 2015 @04:55PM (#10342315)
If I have one piece of advice to give (that you probably already know), it is to have a sound way to measure results and ensure that you are improving. Usually that involves custom tooling that falls within the integration test layer. So if 'inspecting the differences' is a reproducible and statistically reliable way to know that you are good - then you probably are!