In my experience, I always write a clear, idiomic C code along with my intrinsics-based vectorized function, alo png with comment on tricky part (such as using packus for clamping thing to 0-255). You got clear code, and also optimized code (you might want C code anyway for pre-sse2/mmx and non-x86). Downside is that you are to maintained multiple copy of code, but if you have the version optimized for each sse2/sse3/sse4.1/avx2 anyway it is not really that more hassle.
I think the best solution is to be able to make some kind of annotation, or other way of declaration, on a function that says "this function should be no worse than this". In Scala's case, for example tailrec. I'm unfortunately having a hard time with coming up with other, specific examples, but the gist of it is that the compiler either manages to do all the work on the function itself and the functions that that function calls, or errors out and reports what it couldn't do. Ideally I would want to make 10 functions which are all pure and referentially transparent, call all those functions from a top function with some kind of annotation that gives some demands with regards to optimizations, and then have that function be transformed to a single, efficient, fused loop with no allocations or intermediary values that are unnecessary. But like I mentioned, the hard part seems to be in actually specifying what your demands are.