Then all of a sudden you have terabytes of data updating a model faster than you can keep up with it, and after a couple months, starting from scratch to get the same model doesn't seem plausible anymore. Two years of production time and you have a honking mess.
Also it is easy to fall into the trap of prototype-to-production, when you've developed a nice looking demo that suddenly has all of the real customers and none of the engineering rigor.
It is an interesting time to be in software :)
I'm not disagreeing, I'm just wondering if there's somehow a middle ground. What do you think? Can you share any experiences or tips?
I am running my own company at the moment, and my main dataset is about 2.5 TB, it's a very skinny table (it's actually a 191,000,000 x 6,000,000 matrix). It takes too long to iterate the whole dataset, so when I'm training my machine learning classifier I subsample the large dataset then compare previous model results to the new model, and inspect the differences.
Models in production that are total black boxes. There's no way to deterministically recreate them.
But I treated it as an opportunity --- was able to teach some engineering to the data science guys and they taught me some data science :)