It's a shifting target though.

Let's take your example of a program with a lot of I/O. A straightforward way to optimize that is to find a way to reduce the number of I/O operations you do.

And once you do that, the bottleneck shifts. You're spending less time in I/O, both in an absolute sense and relative sense. So you might run into a new non-I/O new bottleneck that was just drowned out in the noise before. So you optimize that ...

And sometimes this goes on for many iteration cycles and you end up with a 100-1000x performance improvement.