Net-net, data and code need to be structured for AVX to achieve the potential performance gains, and that's 80% of the work.
Once you structure the data and code for AVX, yes you can use regular C statements, then experiment with optimization flags until the compiler generates the intended instructions (and hasn't introduced excessive register spills). But its hard to see how that's any easier than using the intrinsics.
You are correct that, generally speaking, most STL heavy code would be hard to vectorize and unlikely to gain much advantage. (Plus there are the valarray misadventures). You will sometimes see clang and gcc vectorize std::vector if the code is simple enough, and they can assume strict aliasing. Intel's compiler has historically been less aggressive about assuming strict aliasing.
Various proposals are working through the standard committee to add explicit support for SIMD programming. E.g. if something like http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n418... were to be standardized we could write matrix multiply explicitly as:
using SomeVec = Vector<T>
for (size_t i=0; i<n; ++i) {
for (size_t j=0; k<n; j+=SomeVec::size()) {
SomeVec c_ij = A[i][0] * SomeVec(&B[0],j, Aligned);
for (size_t k = 1; k < n; ++k) {
c_ij += A[i][k] * SomeVec(&N[k][j], Aligned);
}
c_ij.store(&C[i][j], Aligned);
}
}
For my own work on vector languages and compilers I've had an easier time of it since they have been designed to enable simpler SIMD code generation.
That said, Intel have announced the emergency "Knights Mill" processor jammed into the roundmap between KNL and Knights Hill. It's specifically targeted at deep learning workloads and one might expect FP16 support. They had a bullet point suggesting 'variable' precision too. I would guess that means Williamson style variable fixed point. (I also guess that the Nervena "flexpoint" is a trademarked variant of it).
I assume the FPGA inference card supports fp16. And Lake Crest (the first Nervena chip sampling next year) will support flex point of course. I would expect subsequent Xeon / Lake Crest successor integrations to do the same.
Fun times..
Aside on the compiler work -- I think it's not that hard to emit this instruction at least for GEMM style kernels where it's relatively obvious.