In my experience, cache matters a lot and compilers aren't smart enough to figure out what parts of the code are time-critical and what parts aren't, so they just use the fastest (and sometimes significantly bigger) instruction sequences everywhere. Optimising non-critical-path code by making it smaller even if it's somewhat slower can actually make the program as a whole faster, since the cache misses decrease.