For there to be a media to express their ideas, you need to force some people - private actors, not government - to provide them with such a media. For most people, this is highly unacceptable, because it makes them complicit in spreading this kind of speech. It's one thing to accept that the speech itself should be legal and not prosecuted, and quite another to assist (even if involuntarily) in spreading it.

So, what's more important: the right of Nazis to free speech, or the right of everyone else to freedom of association?