Well, historically, yes. But in my dictionary, bigoted means "obstinately or unreasonably attached to a belief, opinion, or faction, in particular prejudiced against or antagonistic toward a person or people on the basis of their membership of a particular group". So you can certainly be bigoted against the IRS and its employees. No need for the "oppressed or marginalized" part.