> Not sure if any of the test-driven development people have thought about this, but the same principle also applies there: if you debug and fix the code until it passes the tests, you can overfit the tests. It's no longer a good measure for code quality, once it has been explicitly optimized. You'd need new, previously unseen test cases. It would be an obvious mistake in machine learning to simply add failed test examples into the training set and rejoyce that the new model can now deal with those previously difficult cases.

This is a great point, and it articulates some challenges that I’ve seen with test-driven development.

I would say that if you can write tests that fully specify the desired functionality, rather than merely check a few possible inputs, it’s less of an issue. This is a reason to try to build things with less knobs to twiddle, so the space of inputs has lower dimensionality.