"You aren't harmed because you could theoretically have my politics and priorities" is a take.

Dissent from your politics does not require a lack of empathy, and having your politics doesn't automatically mean having it.