I've recently worked on optimizing a piece of integer-based AVX2 code which was running in a tight loop (dot product of 128-dimensional vectors with 8-bit components).

I inspected the assembly generated by MSVC17, and it did a really good job; no extra loads or stores or register spills. It was "as if" hand-written.

> In my experience the compiler does a pretty poor job at this.

A year ago I've worked with a guy optimizing ffmpeg's PRORES encoder; he also used AVX2 intrinsics. The result? 45% speed improvement (gcc) w/o any of the fuss of hand-coded assembly. [1] Sadly, the ffmpeg team suffers from the same prejudices as you, and won't accept intrinsics-based code, even though their "policy" is to accept all patches giving improvement of just a couple of percent points.

EDIT: Ironically, they'd most probably accept the output of gcc -S og forfeit future optimizations implemented in gcc and other compilers.

It is as if experienced programmers are forever stuck with their past experiences and won't ever look back.

[1] https://ffmpeg.org/pipermail/ffmpeg-devel/2016-May/194575.ht...

> It is as if experienced programmers are forever stuck with their past experiences and won't ever look back.

This is totally the case. IME, anything that makes them step back from how they would normally solve the problem and rethink other approaches is of benefit. It is good if you can frame that as an appeal to their ego, 'cos people get defensive quickly. Sometimes they just feel out of their depth with the new and instinctively push back.

Ha, now I'm closely working with a "junior" developer and it's awesome. I try to teach him stuff underway, but usually let him find solutions to problems on his own (within boundaries). Sometimes he digs out really nice stuff that I note down and eventually add to my toolbox. It's almost like having my own research team :)

And for a counterpoint that it's still the case that compilers can produce horrible codegen from intrinsics in 2017, this entire thread: http://www.realworldtech.com/forum/?threadid=166719

Oh. He used 14 registers to manually unroll the inner loop. IME, on modern architectures, manual unrolling doesn't win you anything (benchmarked on, among other things, the above-mentioned dotproduct). I wouldn't be surprised if he got close to optimal code by just NOT manually unrolling. Fewer registers used, compiler less confused.

I thought M.Matz recently fixed the register allocation mess, but apparently only the most egregious parts.

See the old official pre-Matz gcc statements on the mess https://gcc.gnu.org/wiki/RegisterAllocation

Register allocation is a solved problem. The optimal solution can be found in polynomial time for SSA code. SIMD is no different.

Register allocation in general is np-complete[1][2]. Having polynomial-time coloring algorithm for SSA doesn't make the allocation optimal -- you still need to "lower" the SSA to proper machine code.

[1] http://www.cs.cmu.edu/afs/cs/academic/class/15745-s07/www/pa... [2] http://pharm.ece.wisc.edu/wddd/2006/papers/wddd_05.pdf

Solved in the papers or in the available products? Can you please write a little more?

Yep, bingo. There's no substitute for understanding exactly what's being emitted in the innermost loops of intensive operations.

I wonder whether any of this matters much in GPU-land though. I haven't kept up with the game, so the young hotshots that programmed AlphaGo might be doing all of that without resorting to low-level-ASM-type sorcery. I wouldn't be surprised if massive parallelism makes these types of gains a rounding error.

AlphaGo as far as I know just uses standard matrix math libraries to do stuff on the GPU.

AlphaGo does not use GPUs but instead the TPUs.

"AlphaGo was powered by TPUs in the matches against Go world champion, Lee Sedol, enabling it to "think" much faster and look farther ahead between moves."

It used to use GPUs for training because I don't think the first gen TPUs could do training.

You are correct, but take a look at more recent compilers. They are much less embarrassing now --especially if you start using vectorcall. http://godbolt.org is really great for experimenting with SIMD with multiple compilers simultaneously.

Their output is quite alright for straightforward algorithms (crypto, checksums, simpler data processing, nothing meaty like video codecs, don't know anything about that). Though it is of course a good idea to always check.

What are the common cases where you care about every cycle, though? In open source stuff, I've really only seen that in crypto code, where they measure using cycles per byte.

In my case it was a video codec. I imagine these days machine learning code gets optimized a lot ...