Let me see if I can remember what Arvid said. This was like 20 years ago so I won’t get it exactly right.

He was comparing to two CPUs sharing data with shared memory. In the shared memory case, each CPU stays on the happy path but sometimes has to do some cache coherence, which may stall the CPU, but that’s the worst that can happen, generally.

But if you try to send a message to another CPU, then you’re asking it to raise an interrupt. That’s not the happy path. CPU will have to stop everything to receive the interrupt and then divert execution to some interrupt gate. Arvind’s point was that every impl of this is going to be much much worse than the worst case of cache coherence.

Every measurement I’ve ever made confirms this.

Most processors are designed with very expensive task switching, but I don't think I'd call that "inherent". A processor could implement an interrupt as a forced branch 200 cycles in the future. Then it's the same speed as a function call with callee-save registers. That's still slower than letting cache hardware manage the whole thing in the background, but it should be as little as nanoseconds slower.

There's a fairly recent study looking into the cost of raising an interrupt. TLDR: the cost on conventional systems is quite high; a significant part of the cost can be eliminated by using a custom kernel; even with a custom kernel, there remains a substantial cost.

https://par.nsf.gov/servlets/purl/10079614

To my knowledge, the remaining cost could be decreased to approximately the same cost as a branch mispredict, but getting there would require changes to the chip hardware and software stack.

> To my knowledge, the remaining cost could be decreased to approximately the same cost as a branch mispredict, but getting there would require changes to the chip hardware and software stack.

Do it even need to be a misprediction?

If you are completely focused on latency then flushing everything else makes sense. But I would think that if you continue execution for now and put a branch instruction into the queue you'd reduce the cost per interrupt even further.

Sorry for the naive question - but what is an example of “send a message to another CPU”? In high level programming languages, isn’t this usually over shared memory as well? Are there constructs to directly “message” a CPU? When might one use it?