Why on earth would you want a RTOS for HPC?

RT is all about privileging latency over throughput, whereas in HPC throughput is everything.

Perhaps not a RTOS per se, but latency is an issue for some HPC applications. Or not latency per se, but unpredictable latency among different nodes. The basic problem is that many large-scale simulations do frequent global barriers. So if one of the nodes is a bit slower because, say, the OS decided to schedule out the application and run ntpd for a fraction of a millisecond before switching back to the application, all the other nodes will wait. The more nodes, the higher probability that something like this happens, and application scalability takes a hit.

The term you want to search for is "OS jitter", lots of papers on that topic.

Perhaps because a lot of tasks are not embarassingly parallel, which means moving data around, which means network interconnects, which means latency matters. Every millisecond delay in forwarding a packet is thousands of instructions stalled.

I'm not saying this is actually a sensible tradeoff for any extant HPC architecture, but it could be. You certainly want the 'OS' that runs the pipeline on your CPU to be real time!

It is a misconception that RT means "lowest possible latency".

It actually means "latency with a known upper bound". The point of it is to be able to give engineering guarantees - the actuators on the rocket engine vanes will move within N milliseconds of the input voltage from the gyroscope module changing; the robot arm motor will be de-energised within N milliseconds of the laser perimeter sensor activating.

Bounding the worst case can often mean making trade-offs in the best case, and that doesn't make sense in HPC because the hard upper bound just isn't necessary in this environment.

True, a lot of tasks are not embarrassingly parallel, as you point out, but most tasks that run on supercomputers are embarassingly parallel. That's why those supercomputers have value in that problem domain in the first place.

It's unlikely that you would, but once you make it through Linux, the BSDs (including macOS), Unix systems (legacy?), Windows and Minix there's just not that much else left that's modern. At least with a RTOS or something designed for embedded systems you have some hope that it's designed to be lightweight.