It's a term against abusers of a certain type of privilege, which is why you generally don't see it applied to black people.