But we aren't better off with JIT in software. x64 processors work because they are doing things dynamically in separate hardware. It is part of how they use ever increasing transistor budgets to speed up serial processing.

Really, at this point the transistor budget is mostly going to bigger caches and more cores.

It's really little short of a miracle that Intel still managed to extract double digit percent improvements in serial code for all of Sandy Bridge, Haswell and Skylake.

It's less of a miracle and more like "larger OoO window, K done"

No, that's not so easy.

You cannot fill that out-of-order window if you mispredict branches. So the branch prediction has gotten better and better, to an absurd level of accuracy.

You also need more execution units, to accommodate more kinds of serial code without stalling the pipeline. In Skylake they also improved the retirement unit, apparently to improve hyperthreading performance.

There are also better prefetchers and decoders, while the decoders also become more complicated because this is good old x86's variable-length encoding, with further complications for x86-64, AVX, and so on.

TLBs also get more complex (there have been two-level TLBs for a while now), though I guess that is part of the "bigger caches" part.

I agree, I was being a little blunt, but it's not necessarily as complicated as people make it out to be. The best branch predictor is once again the perceptron, and it beats current Intel chips on benchmarks. (I would hazard a, guess that Intel uses something like the tage predictor)

JITs do not and cannot have a complete program view.

Whole program compiler can insert JIT-calling code (specializing for common pattern), JIT compiler cannot view whole program.

One of the things JITs can't do is to add and/or remove parallelism from the program (i.e., add/remove OpenMP pragmas).

> JITs can't do is to add and/or remove parallelism from the program

Of course they can. A trivial practical example is the JVM JIT using vector instructions to add parallelism to a program that didn't have any before. But a JIT compiler could also profile code and add thread-level parallelism if it wanted.

For thread-level parallelism to be added you need to perform something akin to Fourier-Motzkin resolution algorithm or Omega test to find out what dependencies program has and has not.

FM is NP-hard. Omega test is imprecise.

And in general you need more complicated checks like aliasing checks. Which are equal to stopping problem, most of the time.

Usually you can't spend that amount of time on EACH INVOCATION of a program.

The techniques you are talking about are for proving that there is parallelism. The beauty of a JIT is that you can speculate that there is parallelism, go ahead, and then sort out the mess later and recompile to sequential if there wasn't.

https://en.wikipedia.org/wiki/Speculative_multithreading

You can use hardware functionality such as TSX to support this.

You can't speculate about parallelism when dealing with shared resource.

It is unbelievable you even suggested it.

TSX won't save you.

> You can't speculate about parallelism when dealing with shared resource

Why do you think that? What do you think the optimistic software transactional memory algorithms are doing if not speculating on parallel access to shared memory?

Because you have O(n) for size of transaction and this O will quickly rise to overflow any buffers you have in hardware, for one.

Because Microsoft tried to add STM to the .Net and get 4x slower code after extensive effort: http://www.mail-archive.com/[email protected]/msg7973...

Should I continue from these two arguments?

You've just said that you 'can't speculate about parallelism when dealing with shared resource', and then gave me an example of a system that did exactly that. Clearly, it is possible to do speculative parallelism of shared resources. We may not yet know how to make it efficient, but it is possible.

From what I gather, STM in .Net heavily relies on the static program analysis.

You may see it as a proof of your point, but I see it as a counterexample - a team of smart people was not able make speculative parallelism efficient even with static analysis. JIT will be even less efficient as it has to keep and account for tracing information and for speculation logs. These two interact in interesting way, I guess.