That conversation puts you into the position of having to prove you're not a *ist/*phobe. At least I suspect that would be the reaction of most people.

I'm not against working with people who are pro-social justice, even if I don't always agree with them. I am against it being forced on me on pain of losing my job.