That conversation puts you into the position of having to prove you're not a *ist/*phobe. At least I suspect that would be the reaction of most people.

I'm not against working with people who are pro-social justice, even if I don't always agree with them. I am against it being forced on me on pain of losing my job.

That conversation sounds like the best way for both parties to get their preferences satisfied.

How would you phrase it?

Personally? "I recognize that women, minorities, people of color, and LGBTQ folks have struggled with systemic discrimination and I'd like to help mitigate that."

Individual responses will vary of course but it's best to be completely honest.