You have kinda placed yourself in that position by trying to draw the line, though. I'm not saying it's a bad thing to do, but you can't both do it, and be ambiguous about where it's at, at the same time.

Also, you know how they say that one should listen to people's experiences, and that tone policing it a way to shut down inconvenient discourse and secure one's stereotypes? That's very much true, but it cuts both ways. If a black guy told you that he's seeing a pattern of dismissive and hostile behavior towards himself from white people, would you consider that a racist observation? I wouldn't, and I'd believe him, too. And that doesn't translate to "white people are bad", either, but there is clearly a cultural aspect in there that cannot be ignored. Such aspects exist in other interactions, though, and the oft-repeated assertion that they are only relevant when the victim belongs to a disadvantaged social group doesn't have any rational basis.