News for hackers. Stuff that matters.
Re:Sandy bridge, Ivy bridge, Haswell and the mess of Intel p... by jandrewrogers on Friday December 26, 2014 @11:12PM (#8801657)
There is a pervasive issue in that most code written today is tacitly designed for the assumptions of microarchitectures from a decade ago, and most code is even older and is rarely rewritten.

Generally speaking, compilers have a limited ability to improve the basic design of code or deviate from the assumptions about hardware embedded in that code. More often than not, taking advantage of different microarchitectures is not just about the code generation, it is also about algorithm design and selection, which falls directly on the programmer. I currently design software that assumes the Haswell microarchitecture. While the code is written in C++ and the high level architecture is generally the same, the lower level idioms are different because they reflect the strength of the microarchitecture. If the code was for the latest ARM core, it would need to be quite different to be efficient. And I am not even talking about things like AVX2. Idiom changes to match microarchitectures are not "micro-optimizations" in the traditional sense; it will often buy 2x performance improvement overall versus well designed generic code tacitly optimized for another microarchitecture.

The reality is that almost no one writes code for e.g. the Haswell microarchitecture. The idea of writing code for a microarchitecture at the level of, say, C++ does not even enter most programmers' minds. Broadly speaking, a compiler will not allow you to take advantage of advancements in microarchitecture except by accident, which works sometimes but is not efficient in terms of return on the new microarchitecture. Compilers learn some micro-idioms, but most of the macro-idioms escape them (like designing algorithms for the ALU parallelism of a particular microarchitecture).

The last big microarchitecture change for Intel was Nehalem, the original "i7". A lot of code optimization assumptions from prior CPUs went out the window with that microarchitecture. The latest, Haswell, added more usable ALU parallelism and the BMI2 extensions, which are quite useful if you know how to exploit them but nothing Earth-shattering. AVX2 was nice, but too limited to have much value for most normal code.

That said, I am pretty excited about AVX-512, which will be a major microarchitecture extension once that it is available. The giant caveat is that no one will be able to really take advantage of it without, once again, redesigning their code for that microarchitecture. A compiler won't be able to do that for you, which has been the real stumbling block for the adoption of advanced features in new microarchitectures. What constitutes an optimal algorithm or data structure is microarchitecture dependent.

Re:Sandy bridge, Ivy bridge, Haswell and the mess of Intel p... by WallWextra on Saturday December 27, 2014 @02:08AM (#8801936)
It's worth pointing out that Sandy Bridge was a massive microarchitectural change from its predecessors, using a totally different out-of-order engine.

I don't know why the performance characteristics are so similar. I guess once you have a big fat memory subsystem on-die, everything else is diminishing returns?

Re:Sandy bridge, Ivy bridge, Haswell and the mess of Intel p... by nkurz on Saturday December 27, 2014 @02:41AM (#8801992)
I don't know why the performance characteristics are so similar. I guess once you have a big fat memory subsystem on-die, everything else is diminishing returns?

I don't find that to be true. As Andrew is saying, the performance characteristics only look similar because the software is rarely written to optimize performance on modern processors. I have a 5-year old overclocked Nehalem that can beat most current Haswell's on single-threaded code of this sort just because of the higher clock speed. But for algorithms designed for the capabilities of the newer instruction sets, the different generations really start to distinguish themselves.

Instead of ~5% improvements between Intel generations on standard benchmarks, you can sometimes get 50% or more for architecture specific algorithms. From Nehalem to Sandy Bridge to Haswell, the maximum per-cycle reads from L1 has gone from 16B to 32B to 64B. This means that approaches that would have been silly 5 years ago (like 16KB lookup tables from which you need to read 32B every cycle to sustain throughput) can be practical now.

Re:Sandy bridge, Ivy bridge, Haswell and the mess of Intel p... by Tuna-Fish on Saturday December 27, 2014 @10:51AM (#8802854)
> I don't know why the performance characteristics are so similar. I guess once you have a big fat memory subsystem on-die, everything else is diminishing returns?

The first-gen PRF design was built to have the throughput of the old one, while having much gentler design limits and lower power consumption. Haswell expanded the execution engine to 8-wide (4xALU, 2x load agu, 1x store agu, 1x store data), something the older design couldn't have done.

Re:Sandy bridge, Ivy bridge, Haswell and the mess of Intel p... by userbinator on Saturday December 27, 2014 @08:45AM (#8802517)
The last big microarchitecture change for Intel was Nehalem, the original "i7". A lot of code optimization assumptions from prior CPUs went out the window with that microarchitecture.

I'd say the P6-NetBurst change was the biggest, and in many ways it was also a failure specifically because of its extremely different performance characteristics; it was far more "RISCier" than its predecessors or successors, and instruction sequences that were fast on the P6 became much slower (e.g. shifts). The long pipeline made branch penalties increase. It tried to compensate with high clock frequency, but that didn't always work. Existing software had to be recompiled to get much improvement (and compilers updated accordingly), something that a lot of people just didn't want to/couldn't do, for various reasons. Fortunately the change to P6-based Core brought back with it the more predictable performance characteristics of the P6, and its successors have been more incremental improvements on that than a radical new direction.

it will often buy 2x performance improvement overall versus well designed generic code tacitly optimized for another microarchitecture

2x is a big gap, but it's still quite small compared to the 5-10x (or more) between NetBurst and P6 on the same code.

Re:Sandy bridge, Ivy bridge, Haswell and the mess of Intel p... by hga on Saturday December 27, 2014 @09:11AM (#8802582)
[Netburst] tried to compensate with high clock frequency, , but that didn't always work.

In part because they grossly misjudged what speeds they could achieve. It was sold on the premise it could be driven to 10 GHz (maybe 5 GHz in some accounts; it was viewed as a "mar(keting)architecture that could boast high raw speeds), but they had extreme difficultly achieving 3.8 GHz, and as I recall the systems of the day weren't really geared to dissipating the heat produced.

Re:Sandy bridge, Ivy bridge, Haswell and the mess of Intel p... by hdevalence on Friday December 26, 2014 @11:55PM (#8801740)
Do you have any recommendations on material to read on the subject of choosing optimal data structures / algorithms for a particular microarchitecture?
Re:Sandy bridge, Ivy bridge, Haswell and the mess of Intel p... by jandrewrogers on Saturday December 27, 2014 @12:55AM (#8801838)
Most of this follows from understanding (1) what kinds of things a modern core can do simultaneously and (2) how long it takes a core to do those things. On top of this, there are many additional rules regarding memory and cache access behavior.

One of the most useful references for me is Agner Fog's instruction tables, which are updated regularly and you can download for free. They are genuinely an excellent resource. If you roughly understand how C or C++ is translated into machine instructions (not a big leap) then you can understand how bits of code will interact with the microarchitecture and what surrounding instructions can be executed concurrently in the same clock cycle. Basically you can have multiple 'threads' of execution running in a single core at the same time per clock cycle on a modern CPU. Good CPUs will try to do this for you but have limited ability to find this concurrency if you do not make it 'obvious' to the CPU with code idioms that make it impossible to miss. The amount of concurrency you can extract depends on the specific instructions your code is being translated into. Haswell has 4 ALUs per core but each ALU has different capabilities in terms of the subset of instructions it can execute; for example, some basic instructions can only be scheduled on two of the ALUs.

http://agner.org/optimize/instruction_tables.pdf

Additionally, the architecture docs from CPU vendors is always insightful about the topology of the silicon, how everything is wired together, and the tradeoffs that are never mentioned in high-level marketing docs.

Re:Sandy bridge, Ivy bridge, Haswell and the mess of Intel p... by DSingularity on Saturday December 27, 2014 @10:28AM (#8802787)
Good post. I never thought of potential for macro idioms. Are you talking purely about loops or other things like synchronization or memory model related stuff? Can you give a few examples of these macro idioms?
Re:Sandy bridge, Ivy bridge, Haswell and the mess of Intel p... by whyrusleeping on Saturday December 27, 2014 @01:16PM (#8803360)
Could you write a blog post about this? I would love to hear more about the work you do and the considerations you have to take into account between micro-architectures.
Re:Sandy bridge, Ivy bridge, Haswell and the mess of Intel p... by rwallace on Saturday December 27, 2014 @12:40PM (#8803227)
Do you have any examples of this handy? I have seen a lot of cases of idioms and algorithm design changing to match the exact CPU model. Approximate CPU generation, sure: try hard to cut down on memory traffic - but that's true for every CPU nowadays. Misaligned accesses aren't expensive anymore - but there's almost never any good reason to do them anyway. What else is there?