The idea is that you don't measure only once. You measure thousands, tens of thousands of times and more. Then any difference however small it will be also multiplied thousand times and it will be possible to measure it. Even if you will have other random processes such as OS scheduler, network latency, etc - it will be pretty obvious after thousands of measurements which ones to ignore. For example, check the histogram graphs in this paper: https://mlq.me/download/netspectre.pdf
With a timing attack, you can iterate the bytes of the MAC, itself probably based on a cryptographic hash like HMAC(sha256), until you get something that gets to the second stage.
You reduce MAC-forging complexity from 2 to the power of the number of MAC bits, to 256 times the number of bytes in the MAC.
This has been the conventional thinking of software engineers for the past few decades. It turns out to be completely wrong.
I mean who would have thought measuring memory access latency could turn speculation into an arbitrary kernel read gadget yet here we are.
It's hard to imagine the attack you describe being viable.