Re:A Way to Detect Bias
by yummyfajitas on Sunday November 01, 2015 @02:25AM
(#10485714)
No, it doesn't assume or require equality. The p-value you compute for this test will be proportional to the smaller of the sample sizes.
Mins will fail if you conspire to cheat the test, it's true. Very few statistical tests stand up to conspiracy theories.
Another reason the use of mins here is not helpful, is that adding one equally awful accepted candidate to group A and B would then remove whatever bias there was according to the test, which is not what we want the test to indicate.
The idea by PG is a rough rule of thumb and breaks down trivially - suppose VC fund X were to accept all candidates, but group A was worse than B, the test would falsely imply that the fund was biased.
It's unfortunate the idea was dressed up in statistical persiflage because it isn't rigorous -- it's a rough guideline. To make it rigorous wold be very hard: either the abilities of the candidate populations would have to be measured very closely (unrealistic), or a more scientific experiment conducted (A-B test where candidates from each group are included or excluded opposite to the prior decision, which would need big groups).