Thats exactly what Karl Popper says. We should listen to them, aslong as society is strong enough to defend a tolerant society against intolerance.

The question is: When do we reach the point where the defence starts to break?

You could argue that that point is reached when terrorists storm the capitol.

I don't know if we reached that point already. What I do know: If we wait for too long, we reach a state of no return.