It's not solely being used as a class signal; otherwise it would be used to refer to socially similar white people.

I think it's completely uncontroversial to say that usage of the word always has a racial component.

Which statements with racial components are also racist is not a discussion that I believe can be usefully debated on a forum like HN.