Inlining SIMD functions is typically not possible in most applications unless you're compiling a dozen versions, one for each possible CPU, since different CPUs will use different SIMD functions that are optimized for their performance characteristics.
Nevermind the fact that intrinsics generally give atrocious performance compared to properly-written code, as your average 3-year-old can probably allocate registers better than gcc.
> Inlining SIMD functions is typically not possible in most applications unless you're compiling a dozen versions, one for each possible CPU, since different CPUs will use different SIMD functions that are optimized for their performance characteristics.
I mainly care about games when it comes to high-performance programming, so this is not an issue for me. Consoles are obviously fixed platforms, and on PC any game is likely to have high enough performance requirements across the board that you can compile your executable against a SSE3 min-spec target without excluding users because of that.
> Nevermind the fact that intrinsics generally give atrocious performance, as your average 3-year-old can probably allocate registers better than gcc.
I'm glad we agree that GCC is a pile of shit (and not just in this respect). MSVC's performance with intrinsics has also gone up and down with versions but by now it's pretty solid.
The instruction set available is often not the most relevant thing. In many cases, the optimal code will differ wildly between CPUs regardless of the available instruction sets. Let's look at some quick examples of performance characteristics:
Athlon 64: Slow SSE2 unit, MMX is often faster than SSE2 for many functions.
Phenom: Very fast SSE2 unit, but missing SSSE3 support, so can't take use the same optimized functions in many cases as the Core 2 and above. Significantly higher instruction latency than Core 2, so needs significantly more pipelining.
Core 2 Conroe: SSE shuffle operations (punpck, etc) are excruciatingly slow: 4/2 for shuffles that use both arguments as input data (e.g. punpckldq), 2/2 for those that use only one argument (e.g. pshufb). Cacheline-split loads are extraordinarily painful (equivalent to an L1 cache miss, or ~14 cycles).
Core 2 Penryn: Same as Conroe, but shuffles are fast now (1/1 for everything basically).
Nehalem: Cacheline-split loads are cheap now (2 cycles), and shuffles have doubled throughput compared to Penryn (1/0.5).
This doesn't even get into the more instruction-specific messiness, like how some CPUs like movddup and movhlps on integer data while others don't. It's not uncommon to have 3 or 4 assembly functions just to cover the latest CPUs -- not even counting old ones. Fortunately for our sanity, these are usually templated from a single function, with small changes to the relevant areas created via macros.
Most programs aren't like codecs or even like games but could still make great gains from a properly administered dose of SIMD intrinsics.
Many of the limitations that you discuss are real, but we've got strategies to avoid them.
We do compile a dozen versions, which is horrible, but there it is.
We get peak performance out of code using intrinsics. By 'peak', I mean close to the limits that can be achieved given the latency of the critical path and the number of operations that need to be carried out in total divided by the ports that can execute them.
In order to persuade gcc to get register allocation and scheduling right, we typically have to program in a style that closely resembles assembly, and a fair bit of our critical code is automatically generated and hand-unrolled/pipelined/scheduled. This is still easier than coding everything in assembly, especially when something goes wrong.
Large quantities of unrolling, and careful management of potential problems like aliasing, typically gets modern x86 what it needs to be able to overcome the problems of gcc. :-) Quite often the asm generated looks like shit but I have noticed a number of cases where improving the asm to be more sane actually reduces performance, often due to the fact that a modern x86 processor is doing weird things with the resulting code.
That SO question actually did a pretty good job at illustrating your point, because most people assumed that code that works well on their CPU would work well on a GPU, without stopping to think about the architectural differences.
The point is that the 'simple and optimal instruction sequence' I mentioned in my top post has a 1:1 equivalent on every modern CPU and GPU.
Using SIMD intrinsics is often preferable. For set-in-stone inner loops, the extra performance you can wring out by hand-writing in assembly may be worth it, especially on compilers with shitty support for intrinsics. For SIMD-based leaf functions that are going to be inlined in many contexts, intrinsics have the advantage since the compiler can schedule and register allocate the instructions piecewise rather than treating them as an indivisible block with hard-wired registers (GCC-style clobber lists are supposed to help the compiler do that with inline assembly, but it sucks in practice).
My opinion is that every programmer should learn enough assembly to fluently read the code generated by their compiler, but most of them should not be writing it. Debugging is the number one pay-off, but keeping tabs on what the compiler's optimizer does with your C code is a close second.
There was a good illustration of this last week on Hacker News. Someone submitted a StackOverflow thread where they were competitively coding a fast fixed-size sort based on sorting networks. The tentative winner at the time was using some god awful hand-cooked branchless code for min/max that was completely defeating the compiler's attempts at generating the simple and optimal cmp r0, r1; mov r2, r1; cmovg r1, r0; cmovg r0, r2 instruction sequence. That would have instantly obvious if they had the habit of reading their compiler's assembly. The result is that they were leaving a ~3x performance gain on the floor.