I think it's fair to assume any significant HPC solution will have to be recompiled to target its runtime environment. Squeezing out every GFLOPS of a part is normal when you get an expensive specialized computer.

In that context, x86 compatibility is less of an issue than the quality of the compilers. As long as you are easier to program than a GPU, you are good.

I wish this were true, but it's not. Multi-million-dollar contracts have been scuttled because a given customer runs codes that they receive as binaries from a vendor, closed-source, and Company A bid Processor X, but the codes only run on Processor Y...

That's interesting. I'd assume a market so performance driven would compete such vendors out of existence.

The market is performance-driven because it is task-driven. Performance as a requirement, in other words, is a byproduct of the mission. If your task requires software that someone else holds the keys to, you have no choice but to mold the rest of your environment to fit that software.

Thankfully this sort of situation is becoming less common over time, but we're not all the way done yet.

It's a big relief to know my assumptions will, eventually, be true... :-)