It would be equally fair to say that the same die uses less than 5% of it's space for the integer ALUs. It could also be said that the decode unit is so complex that it takes as much die space as all the integer ports put together.

SRAM may use most of the die, but it isn't using huge amounts of power most of the time as it's static. Those big SIMD units are also powered off when not used (no doubt it's the same with the other ALUs).

In contrast, the entire 5% of the die that makes up the decoder never power gates and is always running flat out. By size it's the same as leaving all your integer cores running full-out all the time.

The bigger issue seems to be that neither AMD nor Intel have been able to make those units much wider over the years which puts a hard limit on total throughput (they've hinted that the power cost on widening the decoder is very big which would mean loads of power used for diminishing returns).