Yes. Things can flow the other way as well.
Did black kids pick some fights with me in school because I was white? Yes. It's a real thing that happens.
It's a fallacy, of course, to dismiss a belief just because some of its believers are assholes, even if they're being assholes in the name of that cause. I mean really, no group or belief on this Earth could ever pass that criteria.
The point that you're [ missing | skirting ] though is many folks use the existence of privilege to ignore/bully/hate (eg, my YT link above, the white professor who got berated on campus, etc etc).