Having it repeatedly drilled into you your entire life from birth that you'll never succeed without the pity and generosity of a few benevolent saviors among a sea of hateful racists who are this this all-powerful/privileged alien race that surrounds you (assuming you manage to dodge all the police bullets flying at you) because the system has been created to keep you down, and therefore any success you do have you owe to those people, would be extremely stressful and demoralizing.

You could be on to something here. I've seen many other factors like education and wealth being adjusted for but not this. I wonder if anyone has attempted to quantify the damage this causes.