Isn't that what confidence intervals are for?

I'd say that's more what cost functions are for, but that is neither here nor there.

I think the biggest takeaways are...

1) Machine learning in large software projects is complex because decisions made by one algorithm can influence the data of other algorithms, creating massive biasing.

2) Simple cross-validation / hold back methodology is limited as we expand what we want machine learning to be able to handle. Reason: big data is too big and "correctness" is difficult to evaluate for things like Q/A systems.

Indeed it is and machine learning not being "real statistics" but ad-hoc methodologies (based on statistics but not being statistics) doesn't give confidence intervals etc.

Plus read the article/presentation. It goes into the issue with great depth and clarity.

If we pick a silly thing like recommendations, they happen in a multidimensional space. Not only do we not know the right metrics, shape of the space or cost functions, our feedback is distorted by the actual recommendations we give!