But to your point - yes of course it's ok to disassociate with someone who is advocating violence or National Socialism, because it can reasonably be inferred that said person isn't rational, or has a twisted sense of morality than makes associating with them damaging to yourself. However, I'm assuming that's not the case here - it had come in vogue in SWJ and intellectual circles to immediately disregard anyone who has an incorrect opinion. Vote republican? You obviously are a terrible human being. Fall in line with the Democrats except you think abortion is wrong? Kill yourself.
Given the very serious discussions in US politics about placing US citizens in concentration camps and refusing them re-entry to the country based on their religion, the question about at what point a party becomes the National Socialists must be a legitimate one to ask. Clearly we're not there yet. But also, clearly there's a nonzero chance that we might find ourselves there very soon.
If you want extremist views to have free reign, then keep avoid them until they form a broad enough coalition of the ignored and disaffected to take power, I guess.