So if our society mistreats two groups, only the group that gets treated worst counts? No. Suckiness based on discrimination is bad, period. It's still sucks, even if you're affluent, actually. It's this constant background corrosive.
How much interaction do you have with the justice system? It's no picnic being poor and dealing with them. How much do you deal with the police? It's no picnic being poor and dealing with police you don't know. Organizations across society are crappy to the socioeconomic disadvantaged, period. Heck, even employment is crappy to the socioeconomic disadvantaged!
The thing to watch out for in the struggle for social justice across history: Letting yourself create a scapegoat. Don't let some sort of salient indelible marker become a proxy for who someone really is as a human being. That's the root cause of the problem in the first place! Just as this goes for the fallacies of "blacks being criminals" or "Asians being good at math" or "Jewish people being business sharks" it also goes for white men being angry "supremacists."