I wonder how efficient a counter-measure would be adding small random delays in every part of code, possibly injected at MIR or LLVM level. It might drown any timing information in random noise.

They will definitely lower the performance, but likely a bit slower and more secure connection process is preferable to a less secure one.