You'll be surprised, but yes - it can be measured. Not even on same computer, but across different computers on Internet, even with all the random latency in the middle.

The idea is that you don't measure only once. You measure thousands, tens of thousands of times and more. Then any difference however small it will be also multiplied thousand times and it will be possible to measure it. Even if you will have other random processes such as OS scheduler, network latency, etc - it will be pretty obvious after thousands of measurements which ones to ignore. For example, check the histogram graphs in this paper: https://mlq.me/download/netspectre.pdf