If you know how long data will take to go to/from memory then you can schedule pretty well.

If you don't know whether some value will hit in the L1 or the L3 cache there's wild variance on how long it'll take so you have to do something else in the meantime. On x64, that's the pipeline and speculation. On a GPU, you swap to another fibre/warp until the memory op finished.

Fundamentally the hardware knows how long the memory access took, the software can only guess how long it will take. That kills effective ahead of time scheduling on most architectures.

Iirc caches mostly exist to help keep pipelines full. You could maybe imagine an architecture with deterministic memory access times from specified regions.

Such architectures exist (e.g. for signal processing). They just aren't good for running general-purpose software.

Itanium sold, barely, and still has people using it. But, as you can imagine, not with general-purpose software.

By the 2nd or 3rd generation, Intel had made some changes that really improved performance a lot. Was the final generation of Itanium the 4th generation?

> You could maybe imagine an architecture with deterministic memory access times from specified regions.

Sure.

But it's very easy to level down. And almost impossible to level up.

> Iirc caches mostly exist to help keep pipelines full

You can find out by disabling the stride prefetcher and observing the 90+% performance regression in all your software!

I don't think building a threaded cpu that switches threads to hide memory latency would be philosophically incompatible with VLIW. I think the later generation itaniums were threaded too.

On x86 SMT gets a bad rap because its never been tremendously effective. It's much more effective in IBM's implementation on Power. It wouldn't be hard to imagine some Itanium SMT monster.

How many threads would it take to hide 100ns of DRAM access latency and what serial performance would we get?

It doesn't have to be hidden completely, there are still caches.

With 2 unstalled threads you effectively halve it (in terms of throughput), 4 unstalled threads you effectively quarter it, etc.

> It wouldn't be hard to imagine some Itanium SMT monster.

Itanium did have SMT

Applications tend to exhibit a lot of dynamic behaviour. Consider any graph munger. I do expect there is an interesting subset of applications, particularly those that have fairly narrow scope and that have been highly optimised, which could be effectively statically scheduled. But for general-purpose computation, I don't buy it.

Perhaps as part of the trend towards increasingly heterogeneous architectures, we'll see big VLIW coprocessors for power efficiency in certain serial workloads (GPUs are massively parallel but little VLIW coprocessors; however, they do have dynamic scheduling).

Itanium also does dynamic scheduling. It's not really VLIW - it is just optimized to decode and schedule a lot of instructions at once.

That’s not what dynamic scheduling means. What you need is to dynamically extract ILP so you can get good performance on single thread applications in light of branch mispredictions and cache misses. We are talking SPECint, not SPECfp nor throughput (threading). x86 chips and Apple Silicon show what you can do here and to do that with IA-64 would have beem harder, not easier.

> It certainly was flawed back then, but compiler theory has improved a LOT since then

Which would be fine if there was a time machine available to send back what we know now to the past. But at the time having sucky compilers for what you want to (try to) accomplish was a bad idea.

The same decision now may be good, but then it was a mistake. It'd be like trying to have a Moon landing project in the 1920s: we got there eventually, but certain things are only possible when the time 'is right'.

compiler technology doesn’t help when cache misses and branches aren’t predictable. That’s the fatal flaw.

Spot on. I think it's also important to point out that "the compiler will fix it" line wasn't new with Itanic. The Multiflow guys (for one) said the same thing for their Trace line of VLIW machines 15 years or so before HP & Intel, and that didn't work so well. So now, 40 years down the line, can the compiler really fix it, or do we acknowledge that maybe some things are theoretically nifty but don't work out in practice and move on? I don't know the answer to that, but I have my suspicions.

Because even if you get compilers to optimize for your current VLIW, that makes it harder to make improvements down the line.

Current way, while arguably pretty wasteful on all the micro-optimizing CPU does on incoming bytecode, allows designers to nearly freely expand hardware to meet the needs without having compilers to produce different code.

The halting problem means that for typical programs, you can't prove control flow. Proving optimization means not only proving control flow, but then knowing which branches get taken most often so you can optimize. There may be a small subset of programs for which this is true, but the rest leave the compiler completely blinded.

OoO execution does an end run around this by examining the program as it runs and adapting to the current reality rather than a simulated reality. This is the same reason a JIT can do optimizations that a compiler cannot do. The ability to look 500-700 instruction into the future to bundle them together into a kind of VLIW dynamically is a very powerful feature.

As to compiler theory, it really isn't that advanced. Our "cutting edge" compilers are doing glorified find-and-replace for their final optimizations (peephole optimization).

Look at SIMD and auto-vectorization. There are so many potential questions the compiler can't prove the answer to that even trivial vectorization that any programmer would identify can't be used by the compiler to the point where the entire area of research has resulted in basically zero improvements in real-world code.

This is the same argument that was being made in the 90s, ironically, that compilers were now good enough that it would work.

The thing is, on regular, array-based codes, it's great. And it was then, too -- the polyhedral approach was all being developed at exactly that time, but maybe it's not clear in hindsight because the terminology hadn't settled yet. Ancourt and Irigoin's "Scanning Polyhedra With Do Loops" was published in 1991. Lots of the unification of loop optimization was labeled affine optimization or described as based on Presburger arithmetic. But that is the technology that they were depending on to make it work.

But most code we run is not Fortran in another guise. The dependences aren't known at compile time. That issue hasn't changed much.

The one change now is that workloads that were called "scientific computing" then are now being run for machine learning. But now it doesn't make sense to run regular, FP-intensive codes on a CPU at all, because of GPUs and ML accelerators. So what's left for CPUs that excel on that workload? I'm not sure there is a niche there.