The problem is that the 'racist' label is so poisonous in our (western-European, north-American) culture that it is being used to silence others, no matter whether they actually are racist or not. It does not take much to be called 'racist', sometimes drinking a glass of milk is enough [1,2].

[1] https://www.huffingtonpost.com/entry/milk-white-supremacy-ra...

[2] https://www.nytimes.com/2018/10/17/us/white-supremacists-sci...