So we musn't support anything that's also supported by Nazis? Then we should oppose vegetarianism and bans on smoking, because Hitler supported them?

Isn't this taking things to a logical extreme?

No, my point is exactly the opposite: choosing the worst possible examples of someone who supports X to argue that's the norm is bad thinking. If you want to play the 'broad generalization' game, then you have to be able to suck it up and admit that on one side of the argument we have neo-Nazis, so if you identify with that side you'll be called a neo-Nazi.