It's really little short of a miracle that Intel still managed to extract double digit percent improvements in serial code for all of Sandy Bridge, Haswell and Skylake.
You cannot fill that out-of-order window if you mispredict branches. So the branch prediction has gotten better and better, to an absurd level of accuracy.
You also need more execution units, to accommodate more kinds of serial code without stalling the pipeline. In Skylake they also improved the retirement unit, apparently to improve hyperthreading performance.
There are also better prefetchers and decoders, while the decoders also become more complicated because this is good old x86's variable-length encoding, with further complications for x86-64, AVX, and so on.
TLBs also get more complex (there have been two-level TLBs for a while now), though I guess that is part of the "bigger caches" part.
Whole program compiler can insert JIT-calling code (specializing for common pattern), JIT compiler cannot view whole program.
One of the things JITs can't do is to add and/or remove parallelism from the program (i.e., add/remove OpenMP pragmas).
Of course they can. A trivial practical example is the JVM JIT using vector instructions to add parallelism to a program that didn't have any before. But a JIT compiler could also profile code and add thread-level parallelism if it wanted.
FM is NP-hard. Omega test is imprecise.
And in general you need more complicated checks like aliasing checks. Which are equal to stopping problem, most of the time.
Usually you can't spend that amount of time on EACH INVOCATION of a program.
https://en.wikipedia.org/wiki/Speculative_multithreading
You can use hardware functionality such as TSX to support this.
It is unbelievable you even suggested it.
TSX won't save you.
Why do you think that? What do you think the optimistic software transactional memory algorithms are doing if not speculating on parallel access to shared memory?
Because Microsoft tried to add STM to the .Net and get 4x slower code after extensive effort: http://www.mail-archive.com/[email protected]/msg7973...
Should I continue from these two arguments?
You may see it as a proof of your point, but I see it as a counterexample - a team of smart people was not able make speculative parallelism efficient even with static analysis. JIT will be even less efficient as it has to keep and account for tracing information and for speculation logs. These two interact in interesting way, I guess.
A large amount of the ILP is, AFAIK, from being able to prefetch accurately (IE guess what memory will be accessed) and guess branch directions correctly.
This is not possible to do as well statically. You actually can do it well dynamically (IE i could make a branch guessing thread and a prefetching thread), but you can't do it fast enough.
In a lot of workloads, most time is spent stalling for memory.
Also note that in VLIW, you have to extract ILP or you are just wasting time (IE are executing NOPs). You end up with a half or 3/4 empty pipeline. So unless you can predict you are going to stall for memory well, you are wasting the pipeline. On the other hand, modern OOO cores can just extract more ILP when possible. IE when stalling for memory, they can figure out what they could be doing otherwise.