IMO, this comes from the ideology that the reason underprivileged groups are underprivileged is because of negative social expectations that have a negative subconscious effect on their performance. I don't find it completely convincing (there is much evidence which refutes it), but there are some small experimental studies which support it. I find it plausible that in a group where the minority is extremely small the effect may be large and significant.