It has occasionally been used in a racist way, yes. In that sense, it's a racist symbol, just as the numbers 12, 13, and 14 are racist symbols.

ADL, who seem to be the self-appointed arbiter of such things, says that most uses are not racist.