I think it's fundamentally a fear that political ideas you don't like will become popular and end up having actual power. If they spread enough, people might vote for them, get laws changed for them, etc. or they might just change their culture in a large scale way that affects your daily life because these "wrong" people might be all around you.

Of course that's pure arrogance and complete disregard for the concept of democracy and respect for other people's ability to think.