These ostensibly "neutral" people are in fact biased by their lifetime of experience and the way they skew in light of this confrontation is very revealing about exactly how. No one is introduced to the idea of, say, microaggressions, institutionalised racism/misogyny etc. as a neutral bystander. They have a stake in it whether they've acknowledged it already or not. No one is "pushed" away; they choose, informed by their experiences. And if someone is "pushed" towards maintaining a status quo that some of the most marginalised people in the world think is unfair, maybe they should think about why.