Although I don't really have a dog in this fight, it's important to note that the contemporary literature in the fields of philosophy, sociology and critical race theory generally considers racism to be involving not only a prejudicial element, but also "acts" (it's important to not restrict racism to acts but also to include ideologies) directed from a position of power. The characteristics (i.e qualitative differences) absolutely matter, along with the history which forms both the perception of what's being said (giving words and actions their meaning) but also may provide context for the action to point towards analysing the act critically.

Regarding a related topic, there's a wonderful paper by Swanson[0] on the use of slurs (in particular racist ones) and the ideologies they hail, and the ideologies hailed absolutely depend on who's using them, in a similar way that the use-mention distinction most people would agree exists.

But there's another point here: you have directly assumed a subjectivist-individualist interpretation of racism (and perhaps even race, for a fresh look at the relation between race and racism see "Racecraft" by Fields and Fields[1]), but while you may account for individual actions of prejudice based on "race", it won't account for structural (going by structure driven through ideology) interpretations of racism.

[0] http://www-personal.umich.edu/~ericsw/research/Swanson,%20Sl...

[1] https://www.versobooks.com/books/1645-racecraft