> There is a weird belief, that's particularly American, that if you're not outwardly hateful towards black people, you're not racist.

I think the underlying assumption is that a "racist" is someone who spouts bigotry 24/7, and that racism is black and white (pardon the pun) rather than a spectrum, where even an otherwise "decent" person can act or say something racist on occasion. This is why the common refrain after a person does a thing that is blatantly racist is "I am not racist - everyone who knows me knows this is not who I am" despite the empirical evidence.