arguably the "sufficiently advanced compiler" (cue joke) has arrived (sadly, post Itanium, Cell) in the form of a popularized LLVM[0], so it's improper to claim failure based on two, aged datapoints.

The flaws of OOO and SpecEx are evident with the overhead required to secure a system (spectre, meltdown) in a nondeterministic computational environment, and there is certainly a power cost to effectively JITting your code on every clock cycle.

As the definition of performance is changing due to the topping out of moore's law and shifting paralellism from amdahl to gustafson, I think there is a real opportunity for non ooo, non specex in th future.

OoO and speculative execution are largely improving performance based on dynamic context that in most real world cases is not available at compile time. They are able to do so much more efficiently than software JITting can due to being implemented in hardware. There is still no sufficiently advanced compiler to make getting rid of them a good strategy for many workloads.

Most of what OoO and speculative execution are doing for performance on modern CPUs is hiding L2 and L3 cache latency. On a modern system running common workloads it's pretty unpredictable when you're going to miss L1 as it's dependent on complex dynamic factors. Cell tried replacing automatically managed caches with explicitly managed on chip memory and that proved very difficult to work with for many problems. There's been little investment in technologies to better use software managed caches since then because no other significant CPU design has tried it. It's not a problem LLVM attempts to address to my knowledge.

Other perf problems are fundamental to the way we structure code. C++ performance advantages come in part from very aggressive inlining but OoO is important when inlining is not practical which is still a lot of the time.

My point is that the dominant software programming paradigm is migrating away from highly dynamic to highly regular. A good example is Machine learning, where for any given pipeline, your matrix sizes are generally going to stay the same. A good compiler can distribute the computation quite well without much trouble, and this code will almost certainly not need SpecEx/OOO (which is why we put them on GPUs and TPUs). Or imagine a billion tiny cores each running a fairly regularly-shaped lambda.

Sure some things like nginx gateways and basic REST routers will have to handle highly dynamic demands with shared tenancy, but the trends seem to me to be away from that. As you say, this is all dependent on the structure of code; and I think our code is moving towards one where the perf advantages won't depend on OoO and specex for many more cases than now.

This might be true for some domains but it's far from true for the performance sensitive domains I'm familiar with - games / VR / realtime rendering. The trend is if anything the opposite there as expectations around scene and simulation complexity are ever increasing.

Actually if you read IBM's research papers on RISC, their PL/8 compiler toolchain was pretty much like how LLVM kind of looks like, just on the 70's.

no doubt, but popularity and timing matters.

> The promise of JITs outperforming AoT compiled code has never really materialized.

Well JITs do actually outperform AoT compiled code today. Java is faster than C in many workloads. Especially large scale server workloads with huge heaps.

Java can allocate/deallocate memory faster than C, and it can compact the heap in the process which improves locality.

I haven't seen this convincingly demonstrated. Can you point to good examples? The few times I've seen concrete claims they are usually comparing Java code with C code that no performance oriented C programmer would actually write. In certain cases Java can allocate memory faster than generic malloc but in practice in many of those cases a competent C or C++ programmer would be using the stack or a custom bump allocator.

In practice it's quite hard to do really meaningful real world performance comparisons because real world code tends to be quite complex and expensive to port to another language in a way that is idiomatic. My general observation is that where performance really matters to the bottom line or where there is a real culture of high performance code C and C++ still dominate however. This is certainly true in the fields I have most experience in and where there are many very performance oriented programmers: games, graphics and VR.