Moore's 1965 paper was about transistor count doubling every 12 months rather than frequency but "Moore's Law" didn't come into use as a term until 1975, by which time Moore was giving shrinking feature sizes equal billing[1]. And Moore himself wrote a memo endorsing a more general use of the term "Moore's Law" for any chip performance metric that doubles regularly. I can't find a copy online but I have it in my old Computer Architecture lecture notes. And performance and scaling were identical as long as Dennard scaling[2] lasted.

[1]http://www.eng.auburn.edu/~agrawvd/COURSE/E7770_Spr07/READ/G...

[2]https://en.wikipedia.org/wiki/Dennard_scaling

In the past, the increase of transistor count and density meant that a CPU could be designed to run at a higher frequency and have cleverer circuitry which allowed it to do more each cycle. This is the source of the misconception that Moore's Law is directly tied to performance.

Looking at a modern Intel chip, the CPU cores take about 30% of the area, with the rest of the transistors spent elsewhere (L3 cache, integrated graphics, system and memory controllers).