The effects of entrenched discrimination in the US fade quite a bit when you hit upper middle class or above. They don't vanish entirely, but you have a lot more of a cushion and generally can avoid the problem areas. You can also fight back legally if necessary.