Not that this is perfect, I've got just such a bug out there right now that simply refuses to be reproduced by anyone once a developer is looking at it, but I sure have far fewer of those than I would without the tests.
I'm talking about hacking the application to pass the test suite, not hacking the test suite. You can often "fix" a failing test by doing
if (condition that failed)
whack the application state so that the test suite will pass.
without understanding what the actual bug were. And it's not always obvious to yourself that this is what you are doing.ssp has the problem, not the test suite
Tuning to benchmarks is not some unique character flaw of mine. When you measure some aspect of people's behavior, they will optimize to that measurement. If the measurement is a boolean PASS/FAIL from the test suite, then they will optimize their behavior to get a PASS.
But the actual desired outcome is not PASS, it is "bug free program".
Nevertheless, you are arguing that because people sometimes program to benchmarks, you are apparently better off without the benchmarks. I say that's nonsense. The solution is to use the benchmarks better. Are we programmers or automatons? (Or managers?) If you're going to be that defeatist about programming you're not going to be able to be a successful programmer under any circumstances, the entire field is a minefield of superficially-appealing optimization opportunities!