My half joke is the optimizations slow down compilers. So people then try to make the compilers faster by increasing the levels of optimization. Which means they have to work harder. So they are slower.