1) A lot of the "a ha" magic really has been figured out in microprocessor arch (e.g. pipelining, superscalar, multi-level caching).
Previously, we were reaping the benefits of process shrinks AND microarch epiphanies. Now, only a slower former and less powerful versions of the latter (e.g. branch predictor tweaks).
2) We're discounting the "performance" that has instead been allocated to power. Up until the P4, we had the benefit of just saying "more power!" in pursuit of performance.
Now, not only do we have a power budget to stay within, but we're actively trying to decrease power draw for the same workload in mobile parts.
So instead of getting something twice as fast, we get something that has battery twice is long (or however the math works out).
We've essentially changed the metric of performance from "calculations per second" to "calculations per watt", which can be useful for mobile (battery life) and data centers (reducing heat and the need for massive air conditioners, plus the electric bill), but less useful for gamers that often need more single-core performance.
DOOM on a 386 was nearly unplayable. On a 486 it was perfect. Quake on a 486 was awful, was great on a Pentium, and was smooth as butter on a Pentium II.
These days, each generation is less than 20% faster in single-core performance. My desktop at home is an i7-3770k. In a couple months, it'll be 6 generations out of date, and yet it's still fast enough for everything I do, even gaming!
My CPU is 6 years old, yet doesn't feel like it. If you were running a 6 year old computer in the year 2000, you'd be running a 133 Mhz Pentium while everyone with a new system at the time would be on a 1 Ghz Pentium III. The performance difference was at least and order of magnitude and definitely noticeable.