Really, at this point the transistor budget is mostly going to bigger caches and more cores.

It's really little short of a miracle that Intel still managed to extract double digit percent improvements in serial code for all of Sandy Bridge, Haswell and Skylake.

It's less of a miracle and more like "larger OoO window, K done"

No, that's not so easy.

You cannot fill that out-of-order window if you mispredict branches. So the branch prediction has gotten better and better, to an absurd level of accuracy.

You also need more execution units, to accommodate more kinds of serial code without stalling the pipeline. In Skylake they also improved the retirement unit, apparently to improve hyperthreading performance.

There are also better prefetchers and decoders, while the decoders also become more complicated because this is good old x86's variable-length encoding, with further complications for x86-64, AVX, and so on.

TLBs also get more complex (there have been two-level TLBs for a while now), though I guess that is part of the "bigger caches" part.

I agree, I was being a little blunt, but it's not necessarily as complicated as people make it out to be. The best branch predictor is once again the perceptron, and it beats current Intel chips on benchmarks. (I would hazard a, guess that Intel uses something like the tage predictor)