It's not. Oppressors typically implies privileged, but the reverse is not true. I am not an oppressor because I am able bodied.

Also, you wouldn't use the word privileged to talk about regimes in places with oppressive governments. You'd use the word oppressive.