Thanks, I'll read this over.
In fact, moving platforms now makes little performance difference.
Interestingly, we are saying the same thing but coming to opposite conclusions. I agree that compiler generated code produces approximately the same performance on Sandy Bridge vs Haswell. But I also know that the performance potential on Haswell can be much higher than that of Sandy Bridge, thus see this as a drop in compiler performance. Whereas you seem to be concluding that because the compiler achieves comparable performance, there has been little hardware improvement across generations.
We decided we care more about compile speed than doing stuff like this.
I've never really understood the emphasis on compile speed --- it seems misguided. Sure, you don't want to wait for the compiler to slowly redo the same suboptimal "optimization" on every recompilation, but shouldn't there be some way to cache the previous "best" so you can start from there? And wouldn't it be more efficient if there was some way that you could specify to the compiler where you want it to spend its time, maybe even offline so it won't hold up everything else? Not just the spectrum from "don't optimize and while you are at it please pass all function variables on the stack" to "do the best you can as long as it doesn't take longer than a millisecond", but "do what it takes and I'll tell you when it's good enough"?[1]
It's almost certainly just a small bug or tweak somewhere.
Perhaps I can try to send you an example the next time I come across a compact example. I'm dubious that increasing the automatization is the right approach --- what I think I want is easier access to "manual mode". And I'm equally dubious that it's going to be an easy fix, but would be wonderful it it was.
As a quick example of the sort of issue, Haswell allows two 32B loads and one 32B store per cycle, and has three address generation ports to support this. Two of the AGP's can be used for stores or loads, and one (Port 7) is only for stores. But Port 7 can only generate "simple" addresses --- fixed offsets from a register.
I've yet to find a way to convince a compiler to generate two loads and a store that can be executed in a single cycle without resorting to assembly. Effectively, the cost of an instruction varies depending on what other instructions are executed the same cycle. They can't be scored independently, and the compiler gets stuck in a "local optimum" rather than finding the "global optimum".
Just the ability to say "No, really, I want both of these to be loop variables, and this one to be incremented, and this one to be decremented, and I want the loop to terminate when the decrement hits zero" would be tremendous. Or a least, I'd like to be able to specify this as a starting point for optimization, and have some guarantee that the compiler won't "simplify" by combining the registers unless the "simplification" actually has better performance.
[1] I feel like Dan Luu's excellent and under-discussed article on Software Testing is very applicable here: http://danluu.com/testing/ Like software testing, compilers seem to have settled into the strange expectation that they should do the same thing over and over again very quickly, rather than doing something once and well.
Large pages (2MB+) can sometimes contribute a nice amount of extra performance, of course depending on access patterns. It can also have negative effect under some circumstances, like some very random access patterns. Gigabyte pages could help there, but support isn't great.
Other thing I've investigated is memory channel interleaving. Local memory seems to be mostly 64 bytes each channel round robin, but I guess it can be more complicated too. NUMA systems seem to be either round robin 4096 byte per NUMA region or all CPU local memory in one multi-gigabyte (?) chunk. Understanding memory interleaving can help balancing the work between different memory channels.
Yes. We now do things like http://drona.csa.iisc.ernet.in/~uday/publications/pluto+.pdf
Which will fully, automatically, parallelize and optimize loop nests to the target architecture, including tiling, interchange, cache blocking, blah blah blah Basically anything you can think of. You can't do this by hand. You can do something by hand, maybe tiling, maybe unrolling, maybe interchange but the math is too complex for you to get it right in the general case and to combine all these things at once. We couldn't do this 20 years ago. We couldn't even approach the performance of most architectures.
We can also vectorize whatever you want. The only hard part is calculating the cost vs benefit.
Past that, now we actually have the opposite problem. We come so close to optimal on most architectures that we can't do much more without using NP complete algorithms instead of heuristics. We can only try to get little niggles here and there where the heuristics get slightly wrong answers. We decided we care more about compile speed than doing stuff like this.
"Very rarely do I find that compiler generated code (Clang, GCC, icc) achieves this level of speedup, and even more rarely do I find that the same code gets optimal performance on both of these generations."
I don't know your code, so i really can't speak to this at all, but it's the opposite experience for us. In fact, moving platforms now makes little performance difference. Getting the compiler up to snuff for that architecture does. If what you say is true, it just tells me you need to spend a small amount of time figuring out why the compiler is doing the wrong thing. It's almost certainly just a small bug or tweak somewhere.