Sure, I don't really disagree with any of that; that this entire "Muslim test" works in the first place is because it substitutes it with a group that is disadvantaged in some way, and it "feels more wrong".

But it's still "punching", whether it's punching "up", "down", or "side-ways" doesn't strike me as especially important. Maybe "punching down" is "more bad", but IMHO both are bad and we shouldn't do either. Yet it's still not only commonly done, but also socially accepted. "stop whining, it's not as bad as [..]" doesn't make it okay (not what you said, but some people have, indeed, this is essentially what the entire "#NotAllMen" meme is).