But we're not talking about speech, we're talking about tolerance. Speech is just the medium. And lack of tolerance is a threat to one's life and property: redlining housing developers in the US did not tolerate black people buying homes in white neighborhoods; Nazis do not tolerate Jews to live. Intolerance is not "just" speech, it is speech as a direct encitement to actions that are intended to have negative consequences for the targeted party.