News for hackers. Stuff that matters.
Re:Machine Learning: The High Interest Credit Card of Techni... by binarymax on Tuesday October 06, 2015 @12:56PM (#10340282)
It is surprising how often we take reproducibility for granted in the software world. With unit tests and continuous integration scenarios it becomes such a fundamental concept that it would be unheard of for developing products without them.

Then all of a sudden you have terabytes of data updating a model faster than you can keep up with it, and after a couple months, starting from scratch to get the same model doesn't seem plausible anymore. Two years of production time and you have a honking mess.

Also it is easy to fall into the trap of prototype-to-production, when you've developed a nice looking demo that suddenly has all of the real customers and none of the engineering rigor.

It is an interesting time to be in software :)

Re:Machine Learning: The High Interest Credit Card of Techni... by malux85 on Tuesday October 06, 2015 @04:11PM (#10341975)
I agree - it's so challenging to keep up with Terabytes and Terabytes of moving data ... but reproducibility is so fundamental to science in general.

I'm not disagreeing, I'm just wondering if there's somehow a middle ground. What do you think? Can you share any experiences or tips?

I am running my own company at the moment, and my main dataset is about 2.5 TB, it's a very skinny table (it's actually a 191,000,000 x 6,000,000 matrix). It takes too long to iterate the whole dataset, so when I'm training my machine learning classifier I subsample the large dataset then compare previous model results to the new model, and inspect the differences.

Re:Machine Learning: The High Interest Credit Card of Techni... by binarymax on Tuesday October 06, 2015 @04:55PM (#10342315)
If I have one piece of advice to give (that you probably already know), it is to have a sound way to measure results and ensure that you are improving. Usually that involves custom tooling that falls within the integration test layer. So if 'inspecting the differences' is a reproducible and statistically reliable way to know that you are good - then you probably are!