When they run around with torches and swastikas, they're Nazis. What's the problem? When someone steals your car, are you also hesitant to call them a thief because "people have been called thieves for bad reasons before"? Yes, and?

But generally, yeah, let's test people for lack of empathy and other things. That would be much more interesting than what particular ideology they dress the underlying issues up with, and it would also address the "shutting people up for voicing things we can't handle because they're correct" crowd that hide under the mantle of social justice and whatnot.