I don't know why the performance characteristics are so similar. I guess once you have a big fat memory subsystem on-die, everything else is diminishing returns?
I don't find that to be true. As Andrew is saying, the performance characteristics only look similar because the software is rarely written to optimize performance on modern processors. I have a 5-year old overclocked Nehalem that can beat most current Haswell's on single-threaded code of this sort just because of the higher clock speed. But for algorithms designed for the capabilities of the newer instruction sets, the different generations really start to distinguish themselves.
Instead of ~5% improvements between Intel generations on standard benchmarks, you can sometimes get 50% or more for architecture specific algorithms. From Nehalem to Sandy Bridge to Haswell, the maximum per-cycle reads from L1 has gone from 16B to 32B to 64B. This means that approaches that would have been silly 5 years ago (like 16KB lookup tables from which you need to read 32B every cycle to sustain throughput) can be practical now.
The first-gen PRF design was built to have the throughput of the old one, while having much gentler design limits and lower power consumption. Haswell expanded the execution engine to 8-wide (4xALU, 2x load agu, 1x store agu, 1x store data), something the older design couldn't have done.
I'd say the P6-NetBurst change was the biggest, and in many ways it was also a failure specifically because of its extremely different performance characteristics; it was far more "RISCier" than its predecessors or successors, and instruction sequences that were fast on the P6 became much slower (e.g. shifts). The long pipeline made branch penalties increase. It tried to compensate with high clock frequency, but that didn't always work. Existing software had to be recompiled to get much improvement (and compilers updated accordingly), something that a lot of people just didn't want to/couldn't do, for various reasons. Fortunately the change to P6-based Core brought back with it the more predictable performance characteristics of the P6, and its successors have been more incremental improvements on that than a radical new direction.
it will often buy 2x performance improvement overall versus well designed generic code tacitly optimized for another microarchitecture
2x is a big gap, but it's still quite small compared to the 5-10x (or more) between NetBurst and P6 on the same code.
In part because they grossly misjudged what speeds they could achieve. It was sold on the premise it could be driven to 10 GHz (maybe 5 GHz in some accounts; it was viewed as a "mar(keting)architecture that could boast high raw speeds), but they had extreme difficultly achieving 3.8 GHz, and as I recall the systems of the day weren't really geared to dissipating the heat produced.
One of the most useful references for me is Agner Fog's instruction tables, which are updated regularly and you can download for free. They are genuinely an excellent resource. If you roughly understand how C or C++ is translated into machine instructions (not a big leap) then you can understand how bits of code will interact with the microarchitecture and what surrounding instructions can be executed concurrently in the same clock cycle. Basically you can have multiple 'threads' of execution running in a single core at the same time per clock cycle on a modern CPU. Good CPUs will try to do this for you but have limited ability to find this concurrency if you do not make it 'obvious' to the CPU with code idioms that make it impossible to miss. The amount of concurrency you can extract depends on the specific instructions your code is being translated into. Haswell has 4 ALUs per core but each ALU has different capabilities in terms of the subset of instructions it can execute; for example, some basic instructions can only be scheduled on two of the ALUs.
http://agner.org/optimize/instruction_tables.pdf
Additionally, the architecture docs from CPU vendors is always insightful about the topology of the silicon, how everything is wired together, and the tradeoffs that are never mentioned in high-level marketing docs.
Generally speaking, compilers have a limited ability to improve the basic design of code or deviate from the assumptions about hardware embedded in that code. More often than not, taking advantage of different microarchitectures is not just about the code generation, it is also about algorithm design and selection, which falls directly on the programmer. I currently design software that assumes the Haswell microarchitecture. While the code is written in C++ and the high level architecture is generally the same, the lower level idioms are different because they reflect the strength of the microarchitecture. If the code was for the latest ARM core, it would need to be quite different to be efficient. And I am not even talking about things like AVX2. Idiom changes to match microarchitectures are not "micro-optimizations" in the traditional sense; it will often buy 2x performance improvement overall versus well designed generic code tacitly optimized for another microarchitecture.
The reality is that almost no one writes code for e.g. the Haswell microarchitecture. The idea of writing code for a microarchitecture at the level of, say, C++ does not even enter most programmers' minds. Broadly speaking, a compiler will not allow you to take advantage of advancements in microarchitecture except by accident, which works sometimes but is not efficient in terms of return on the new microarchitecture. Compilers learn some micro-idioms, but most of the macro-idioms escape them (like designing algorithms for the ALU parallelism of a particular microarchitecture).
The last big microarchitecture change for Intel was Nehalem, the original "i7". A lot of code optimization assumptions from prior CPUs went out the window with that microarchitecture. The latest, Haswell, added more usable ALU parallelism and the BMI2 extensions, which are quite useful if you know how to exploit them but nothing Earth-shattering. AVX2 was nice, but too limited to have much value for most normal code.
That said, I am pretty excited about AVX-512, which will be a major microarchitecture extension once that it is available. The giant caveat is that no one will be able to really take advantage of it without, once again, redesigning their code for that microarchitecture. A compiler won't be able to do that for you, which has been the real stumbling block for the adoption of advanced features in new microarchitectures. What constitutes an optimal algorithm or data structure is microarchitecture dependent.