A few years back I decided to entertain myself by testing how smart today's smart compilers really are when it comes to auto-vectorization.

I had this small and simple C application I'd written years earlier that tried to find inputs whose corresponding MD5 hashes started with certain bytes. It was a good base because it was obviously vectorizable.

At first enabling the vectorizer didn't result in any changes to the binaries. I then correctly guessed that (potentially) calling printf function inside a hot loop might confuse it. After slight refactoring I got the compiler to output SSE instructions, which resulted in a nice 2.5× testing speed over the original (incidentally even without auto-vectorizing the refactored code resulted in faster binaries, which is not all that surprising).

Anyway, I also rewrote the application to use intrinsics. I hadn't used them before myself, but it didn't really take much time at all familiarize myself with them and write the code, and it was indeed quite a bit faster than what the compiler was capable of with resulting binary having 14× speed compared to the original, or over 5× compared to what the compiler could achieve without explicit hints from intrinsics.

Edit: added back a few words I had accidentally removed when rearranging sentences, causing a confusing incomplete sentence. Corrected comparing figures like for like.

I’ve had similar experiences. If I want vectorised code, I just write it myself using intrinsics or assembly. It’s fine if the compiler can autovectorise something I didn’t feel like doing by hand, but I’m not going to rely on heuristic voodoo to get the machine code I want for a hot loop. I wouldn’t mind a slightly nicer wrapper API for the intrinsics, though, something like glsl-sse2[1].

And that’s more or less what I’m planning to do in a programming language I’m working on, actually—if you use a SIMD-compatible array type, the compiler will try to keep it in a vector register, and some operations will be faster (e.g., “+” on two Float32^4 values will compile to an addps) but it’s up to the programmer to use the instructions they actually want, or tell the compiler with a macro “please vectorise this loop or warn me about why you can’t”.

[1] https://github.com/LiraNuna/glsl-sse2

That matches my experience. With auto-vectorization you get some speed-up helping the compiler (not always obvious, often requiring +1 increments, etc.), but for full speed you need to do handwritten SIMD intrinsics. I would like to have at least 50% of the optimal by the compiler, without intrinsics (and using intrinsics for the most critical code).

Would be interesting to see the difference in the compiled assembly.

As most people mentioned, the issue in C/C++ is aliasing hence why the compiler cannot safely generate SIMD instructions. However, having worked in scientific computing for a few years on CFD solvers (written in C++), I have seen a very large advancement in what the compiler can do ( Intel compilers especially ). Our approach is to use OpenMP 4.0 directives as they also give you data scoping and alignment clauses. It's probably the best of both worlds where there is some much needed input from the programmer and then the compiler takes care of the rest. On our benchmark, a compute bound kernel would perform perhaps 10% better with hand tuned intrinsics vs the autovectorized version via OpenMP. And that is mostly because it removes some intermediary loads and stores when using short vectors in the autovectorized version which are usually done from a higher level cache anyway. In terms of portability, well, you can see why going for the OpenMP version is the safer bet. One important aspect to take into account is that for efficient autovectorization, you need to rewrite a lot of your kernels to use a vector programming paradigm (basically use short vectors of SIMD size -- which can be changed at compile time depending on uarch ) and trust me, then the compiler will be in its comfort zone. I've seen the compiler generate as good instructions for conditional branching as with hand tuned intrinsics ( AVX/AVX2 and AVX512 ). It can also do transposition from an AoS data structures to SoA in flight however our hand tuned kernels still outperform these since we know to use the lower latency instructions. From a performance perspective, without SIMD, we lose on average 2-3X in our simulation turn around time but getting our whole solver vectorized was a pretty mammoth task.

"Auto-vectorization is hard. Very hard. "

This varies very heavily depending on the programming language :)

C/C++ is not a language that easily enables one to guarantee things about aliasing or loop dependence.

but you can write it in a way that's conducive to better codegen, at the cost of developer sanity.

Or you can design a language that is built all around it. Halide is pretty interesting, basically being a non-Turing complete DSL that lets you separate algorithm from scheduling:

http://halide-lang.org/

Something like that could perhaps also be useful for auto-vectorization.

This is why I like the SIMT model used in CUDA and OpenCL. Just give me a unified programming model for both SIMD vectors and multicore. Let me program each ALU as a separate thread. This requires CPU makers to wake up and give us branching intrinsics that work like in GPGPU: Just let the branched out ALUs sleep / do no-ops. As far as I understand Intel has done that with AVX-512 for KNL. This should be advertised much more. IMO one of the biggest mistakes with KNC was to only push OpenMP and neglect OpenCL. The promise of getting good performance by just recompiling OpenMP multicore code to KNC was snake oil - you have to rewrite all the vectorization in order to even get close to GPUs of the same generation, let alone GPUs released in subsequent years when Intel's production pipeline was stalled.

> I would love to see smarter compilers, understanding the code, many steps over current hardwired pattern-matching based optimizations.

It'll be interesting when we get to that point. I wonder if it'll be in ML or compilers that "cognition" happens first?

> From my experience, despite auto-vectorization being better than 10 years ago, still is very far for optimizing code properly without lots of tuning

Why do you think this is? Does C being difficult to reason about contribute (e.g. statefulness and not knowing what can modify what)?

I've done some SIMD and other optimisations on Android in C for graphics related algorithms and small changes and hints can make a x10 or more difference so I understand your point.

The problem is usually determining when it's profitable, without any profiling, actually.

Additionally, C/C++ are languages where the default is "anything can alias anything" (restrict, until very recent standards, isn't the panacea people think it is) and worse, it's really easy to end up with loop dependences, as well as non-computable loop trip counts, as well. Don't forget alignment, too!

This means inserting runtime checks.

So, assuming the compiler can reorder the loop to vectorize it (and honestly, with polyhedral optimizations, it almost always can if it's at all possible to do so), the question is: is it worth it to vectorize a loop but have to insert 5-6 runtime checks to test for aliasing/etc.

The answer is usually no.

Vectorization, sadly, is not one of those things where more is always better.

Vectorizing every loop/straight line in a program will generally make things much much slower.

(because now you have limited the execution resources to do the computation :P)