My half joke is the optimizations slow down compilers. So people then try to make the compilers faster by increasing the levels of optimization. Which means they have to work harder. So they are slower.
I guess ideally this should be a tradeoff in the hands of the developer right? I should be able to produce debug builds quickly so that I can iterate quickly, and if I want a super-optimal release build it's fine if it takes longer.
Compilers are hard, compiler bugs can have a devastating effect on one's software, so it's understandable that developers may prefer to take things slow but safe.