If you get jumped by several guys from an oppressed group at that moment in time and in that place they have a lot more power than you do. Just like in code that context exist within the broader global context where you correctly point out that there are power differentials.

I also grew up in the south like you did but I grew up in a trailer park. Kids like you with houses on the bus made fun of kids getting on from the trailer park.

My point is that socioeconomic status is dramatically more predictive of failure or success in the United States than race is. Geographic location is also a huge predictor. If we're going to reduce things down to one dimension we should focus on class.

But it's still "punching", whether it's punching "up", "down", or "side-ways" doesn't strike me as especially important. Maybe "punching down" is "more bad", but IMHO both are bad and we shouldn't do either. Yet it's still not only commonly done, but also socially accepted. "stop whining, it's not as bad as [..]" doesn't make it okay (not what you said, but some people have, indeed, this is essentially what the entire "#NotAllMen" meme is).

I mention this because it's another dimension of context which gets badly overlooked when people discuss divisive issues that are dependent on specific places and histories.