Remember Facebook rejected Brian Acton, and paid billions for that. An oversimplification, I agree, but the larger point is that people tend to dismiss too easily the impact of bad “no hire” decisions, compared to the bad “hire” decisions. Just because you can’t or don’t measure it, doesn’t mean that the impact may not be arbitrarily large.
To borrow from poker: it’s commonly understood that if you’re not “caught bluffing” at least a little, it means you’re not bluffing enough. If your hiring process results in zero bad hires, you are playing it way too safe and I guarantee you are missing out on phenomenal candidates.
Now, this assumes that you're not missing out on great hires; in other words, it assumes that the good hires you miss are at the bottom of your hiring range. Your point might be that this can't be guaranteed, and while that may be true, it's unlikely that any tweaks to a given hiring process are going to bring them in either.