These people are real, and actually causing harm. Or do you argue that it is not the case?
The western Roman Empire fell apart a hundred years after Christianity became the state religion. Some historians blame tensions among christian schisms in Egypt/the Middle East for the byzantine empire losing those regions to the Arab invasion.
(All I’m saying with the above is that the justifications seemed plausible and reasonable to the educated people of the time. Read Pliny’s letter to Trajan seeking counsel for what to do about Christians, for example)
I think the answer is more speech, not less. Any exception you carve out will be abused in the future, based on the history of humankind and the behavior of governing entities throughout.
Take these exceptions to freedom of speech, add to them a codified framework for equity (which some are pushing for), and you're laying the groundwork for a society like that seen in Harrison Bergeron.
Roman Christians refused to worship the emperor, destabilizing the social order, and therefore needed to be thrown to the lions to protect society.
Atheists couldn’t be trusted because they didn’t believe in hell, and therefore would act immorally, and so should be banned from positions of power (this is actually in a couple of US state constitutions!)
(And of course, the same arguments were used in reverse later on as different groups got power)
I won’t shed any tears for 8chan who are a bunch of immoral scum, but I know these same arguments will be deployed to censor religious minorities and others in the future. Hopefully they are less appealing targets.