Throw in changing clock rates based on the temperature of the CPU. If your code efficiently uses the ALUs / cores in the processor, it can get hot and slow itself down.
So, that's the best general wisdom I know. Don't even bother trying to guess where you'll have performance problems. Profile it.
I've seen very unexpected results doing this. In one case, I put the inner loop in a separate function to enable swapping implementations and make it simpler to time. Surprisingly, the overall program ran faster without any other changes. Just pulling the inner loop into a function of its own sped things up.
My best guess was that the compiler was willing to put more effort into optimization on the smaller function. This contradicts common wisdom about function call overhead, and it also shows another level of complexity in the problem: Some compiler optimizations are exponential in their cost, so they punt if you put too much of your code in one block or function.
Also note that I wasn't able to prove this is what happened. It's just my best guess based on what I was able to see and infer.
But things are so complicated nowadays that's doesn't even come close to capturing the performance.
Is this the only way we can get good performance? Is there an alternative paradigm of optimization in hardware or software that would make performance tradeoffs more explicit? I don't know, but the current state of things makes me uneasy because I feel I never understand what's truly going on (and I definitely don't).