How can we even get "portable performance" between architectures like ARM and x86/64, let alone from orthogonal paradigms like CPUs and GPUs?

Intel/AMD processors have complex branch predictors and instruction decoders and SIMD, ARM has conditional execution and some sort of quasi-SIMD, and GPUs have a weird hybrid of SIMD and multi-threaded-single-whatever (execution??) with sorta-kinda shared memory. Not only do these architectures implement memory access differently, the way basic control flow operations (conditionals and loops) work is physically different.

That's not even including architectures like SPARC or specialized DSPs. Or FPGAs, which are a whole different universe.

It's not going to be easy, I agree. Matching hand-optimized hand-written code written by experts is not realistic for single primitives, of course. But getting close may be. It's all about programming at a high enough level that the compiler doesn't have to think too hard about what the program means, so that it can instead spend its time worrying about how to express that in the machine code of the target platform.

Imperative languages are probably hopeless, but declarative languages may work. Functional languages in its current popular incarnations is about as hard to compile intelligently as imperative code, but I think that functional array programming (where bulk operations are primitives) show promise. Consider computing the average of every row of an n*m matrix 'a' (a 2D array). In Haskell-ish syntax, we would write this as:

    map (map (/m)) (map sum a)

Now the compiler really only has to figure out two high-level things: how to apply a function (division by 'm') to every element of an array, and how to sum equally-sized segments of an array. Summing is again an instance of a more primitive bulk operation, reduction, which is parallel in theory, and has known efficient parallel implementations on most hardware. It's also possible to do some fusion for this small example, but my point is that we will need a programming model where the individual statement or expression works at a much higher level, and has much stronger properties, than in most current languages. You still need to teach the compiler about each specific hardware platform you want to target, of course (possibly by automatic learning or autotuning, as in http://www.lift-project.org/ ).