The key assumption there is that your alternate-self ghost-car is following the flow of traffic. Which might be true if the route was a highway; but what if it was, say, a shopping district? (“The other you would have gotten here slower, because they’d have been lured into a Starbucks drive-through on the way.”) Or what if they’re a new driver/have a flat tire, and so are driving slower than the speed limit with cars honking behind them? (“Your other self would have been a teenager who hesitated too long on a left turn at an intersection, and so had to wait an extra light cycle.”)