https://en.wikipedia.org/wiki/File:VIA_Isaiah_Architecture_d...

As you can see, caches are merely half of the chip and half of the rest is about dynamic instruction reordering, branch prediction, register renaming and stuff.

These things have pipelined execution units so they can start a new instruction before the previous one is finished executing, enough duplication to start executing two or three instructions per cycle (sometimes even of the same kind, say two floating point SIMD additions) and logic to schedule instructions wrt data dependencies, not program order, so that instructions which need input data not yet available can wait for a few cycles while other, later instructions are executing.

And all of this has to be done with some degree of appearance of executing instructions serially, so if say some instruction causes a page fault and a jump to the OS fault handling code, the CPU has to cancel all later instructions which may have already finished executing :)

And, btw, this is not in any way specific to x86. POWER, high-end ARM, they all do it.

Maybe Agner's site, in particular his microarchitecture manual, would be a reasonable place to start:

http://www.agner.org/optimize/

There are "software optimization manuals" from CPU vendors, but these may not be particularly novice-friendly. I think I've used Wikipedia at times for general CPU-agnostic concepts, though it has a tendency to use jargon with tittle explanation. Occasionally somebody submits something to HN.

On the lowest level, it may be helpful to know some simple digital circuits (decoders, multiplexers, adders, flip-flops, ...?) just to have an idea of what kind of things can be done in hardware.