> Almost everyone is going to either peek at the data, massage the data, or try the validation and when it fails tweak the model and try again, or any number of shenanigans that no one admits happen - but happen all the time.
While "peeking" at the data is certainly something that happens all the time, usually what happens for even the most rigorous of model-makers is that your model fails "out of domain" -- the past doesn't accurately reflect the future.
Even if you get everything else right, it's incredibly common to fool yourself into believing you're capturing "reality", when in fact you're just making a good fit to past random fluctuations.
I already addressed the point about distribution shift above. You need to assume that the past reflects the future at some point, otherwise you can't justify deploying the model even after the prospective test.
Their point is that while in theory you could blind yourself to the holdout set - that rarely REALLY happens. At least in a truly good way. Almost everyone is going to either peek at the data, massage the data, or try the validation and when it fails tweak the model and try again, or any number of shenanigans that no one admits happen - but happen all the time.
The failure mode that a prospective test would guard against in this case is - you can't peek or massage the future (and if you can, you can retire a billionaire pretty easy and stop worrying about this) - so you can't accidentally fool yourself or others by peeking, massaging, etc. And if you tweak your model, then do another prospective test - then you still can't peek at the future, and you'll have a different set of data, so you can't be overfitting to the old data/p-hacking.
You can do it over and over again, and as long as you don't just say 'all good' on a failed test without doing another different prospective test, you either have a predictive model or you clearly do not.
That is something the traditional validation models cannot do, since they aren't looking for truly never before seen data.
It still doesn't tell you if once in production you will stop fitting and fail over, but it does at least tell you that it fit a never been seen, new set of data at least once. Which is better than the other tests can do by their very nature.