This isn't the claim; the claim is more moderate: Nazis gaining political power, and not necessarily inevitably doing so. Nazis are only an example here; you can substitute any 'wrong' idea to see the analogies, from climate change denial to anti-vaxx to flat earth theories. Further, political power is only the end goal. I think most people would agree that a society where a large percentage of the population consists of (Nazis/anti-vaxxers/climate deniers) who don't vote in themselves would have negative discursive effects socially or politically on those who do vote. The state has power, but it does not have absolute power, and that's a good thing.
For many, some more balancing between the ideals of speech autonomy and democratic equality is strongly justified - see some regulation in Europe for example.
[0] https://plato.stanford.edu/entries/freedom-speech/#DemCitPor