sqlite famously squeezed out a ~40% performance improvement (I think from v3 to v4?) by just combining tons of micro-optimizations of this kind where it wasn't obvious if each change even made an improvement. They measured the performance with cachegrind in order to identify very small improvements that get lost in the normal measurement noise.

Also check fishcooking: http://tests.stockfishchess.org/tests which is how Stockfish became the indisputably strongest chess engine: Everyone can commit patches which are then tested in thousands of games, and if they make Stockfish stronger, they are accepted.

That’s a really awesome idea. Also, maybe a few of the performance boosts in the final combination are mutualistic, but if measured independently would not make a difference.

I would love to read more about this. Do you have a link?

Statistics is a tricky beast. If 3 runs gives you a 95% chance that your change is an improvement, and you use this process on ten different improvements, there's a 40% chance that at least one of them is bogus.

Would love to see a refutation from a downvoter.

I believe you have it right. Odds that all changes are improvements in 10 changes is: (.95)^10 Odds that at least one change is not improvement is: 1-(.95)^10 = 0.40

One could probably make an argument that not all trials are independent, but not sure what else down-voters saw.

It is not a good method to use in that case.