The test may not be sufficient to prove that you have no bias, but it may be good enough to prove that you do. When it does indicate bias, it seems likely to be correct.
To put it another way, if it is 1948 and the only three black people in Major League Baseball are all superstars, then the distribution of baseball skill among black players is extremely unbalanced or there is a lot of bias keeping the average and moderately-better-than-average black players out.
Of course favourable treatment can't make people into superstar startup founders or baseball players (and I'm sure any special treatment afforded to black baseball players in the 1940s was the complete opposite of favourable). But more generally it can make an organisation with fair selection processes look like it sets a higher bar for $MINORITY because it addresses low numbers by being very keen to promote and very reluctant to fire/deselect members of said minority, so these kind of studies still have to be considered with care.
(Of course even if an organisation is proactively treating a minority group favourably after selection doesn't mean that conscious or unconscious biases don't exist in the selection process.)
It seems entirely possible that if your program is "narrow" enough you could exhaust the pool of more successful candidates from a minority group. Of course this would be a lot more plausible if you're biased.
It's also worth noting that an unbiased prediction is not necessarily "fair" in the colloquial sense. For example, I've seen data suggesting that an unbiased prediction of college outcomes would actually penalize black applicants, since black applicants underperform relative to their SATs and college grades. (The person who had this data was very careful not to draw this conclusion in the publication - career limiting move, as they say.)
So a fair selection process which looks only at high school grades/SAT/etc might actually be biased as a statistical decision procedure.
It does mean that maybe monetary earnings or anything else sensitive to later-round bias are not the thing to use to measure candidate performance, at least if you're doing this for the social utility.
Of course, if you're only in it to make money, and you're only in charge of the first round... then you really do want just an unbiased evaluation of the (biased) future earnings prospects. So in that case using raw earnings would be correct...
However, the test may still useful to help confirm bias. If outperformance is observed, you can infer one of 3 things is true:
1) there is bias at initial selection but not after (or at least reduced bias)
2) members of the outperforming group are simply stronger performers (different but still interesting)
3) there is no bias at selection but there are affirmative action effects after the initial selection (not obvious why this would be the case)
I would argue that social responsibility requires YC to take the hit, but bias is the wrong word if they don't.
The importance of relationships to the funding round also plays a role. If you get as far as an IPO, it seems unlikely that the stock-buying public is going to stay away because the founders are female/black/etc.
Their basic premise is wrong, if bias continues to exist after the selection event in question.
For example, if YC had (hypothetically) a real bias against black or women entrepreneurs, it is almost certain that future funding rounds, as well as all possible exit scenarios, would exhibit very much of the same bias.
In which case, the future "performance" of those candidates would be poor, and by PG's definition unbiased even though the only meaningful result is that YC is no more biased than subsequent performance evaluations.