There are some optimizations that make a big difference everywhere:
- consolidating pure function calls. This can save arbitrary amounts of time. Classic example is strlen being called many times. I know you're not asking about inner loops, but this can change the complexity of loops which is a big big win.
- mem2reg. Kind of self explanatory. Registers are fast but we write C and C++ using addressable objects. Most compilers make a decent effort here even with optimizations turned off.
- global variable numbering. This allows loads and stores to be removed/moved. Often prevents cache misses or puts them off until the end of a function when it doesn't block execution.
- strength reduction. Turning your divisions into shifts. Turning your && into &. Etc. It is not unusual for these peephole changes to save 10s of cycles per instruction replaced.
These are also really fast optimizations (mem2reg can be extremely slow if "optimal", but the heuristic versions that everyone uses are quick).
If you know you won't care, you can mark the function as cold. That said, the compiler might ignore you and decide you cannot possibly really want to disable something like mem2reg.
Besides, little things add up even outside of tight loops. Slow code in rarely used areas is still noticeable when that rare thing happens.
Is that really true? I'd have thought most code outside inner loops benefits almost negligibly from optimization.