It's a serious problem in compiler optimization. In the ideal world, a program with loop unrolling and vectorization should always be faster, but in practice, the bigger binary size induces cache misses and may produce a slower problem. Time goes by, and compilers keep adding increasingly complex techniques based on space/time tradeoffs, making the problem trickier.

Historically, under some circumstances, a problem compiled with -Os, may paradoxically be faster than -O2, and -O2 may be faster than -O3. I think the situation is getting better now, and the optimizing level mostly works as advertised. But there's still some problems in automatic loop vectorization.