That's somewhat hard to answer because I've been accumulating knowledge from many sources over many years, and it started with some dead-tree book from the '90s :)

Maybe Agner's site, in particular his microarchitecture manual, would be a reasonable place to start:

http://www.agner.org/optimize/

There are "software optimization manuals" from CPU vendors, but these may not be particularly novice-friendly. I think I've used Wikipedia at times for general CPU-agnostic concepts, though it has a tendency to use jargon with tittle explanation. Occasionally somebody submits something to HN.

On the lowest level, it may be helpful to know some simple digital circuits (decoders, multiplexers, adders, flip-flops, ...?) just to have an idea of what kind of things can be done in hardware.

Many CS departments teach from Patterson and Hennessey, Computer Organization and Design for intro, Computer Architecture: A Quantitative Approach for advanced.