Of course favourable treatment can't make people into superstar startup founders or baseball players (and I'm sure any special treatment afforded to black baseball players in the 1940s was the complete opposite of favourable). But more generally it can make an organisation with fair selection processes look like it sets a higher bar for $MINORITY because it addresses low numbers by being very keen to promote and very reluctant to fire/deselect members of said minority, so these kind of studies still have to be considered with care.
(Of course even if an organisation is proactively treating a minority group favourably after selection doesn't mean that conscious or unconscious biases don't exist in the selection process.)
It seems entirely possible that if your program is "narrow" enough you could exhaust the pool of more successful candidates from a minority group. Of course this would be a lot more plausible if you're biased.
The test may not be sufficient to prove that you have no bias, but it may be good enough to prove that you do. When it does indicate bias, it seems likely to be correct.
To put it another way, if it is 1948 and the only three black people in Major League Baseball are all superstars, then the distribution of baseball skill among black players is extremely unbalanced or there is a lot of bias keeping the average and moderately-better-than-average black players out.