>This is an assumption they make about me based on my race, and which some use against me (thinking me privileged in relation to school system).

Conversely, they might use it FOR you. They might assume you're smarter because you're white and went to a better school, and give you that promotion or that job or that loan.