In my experience the case is exactly the opposite. The term "racism" is so overloaded at this point that you can call someone a racist if they've mentioned anything even tangentially related to race. It's an insult more than a descriptor.