The trouble comes when "speech" is arbitrarily defined and/or enforced to suit a narrative. The initial targets are always the lesser-favored extreme cases in order to have the least amount of people disagree with it. Then when the people are comfortable having them define what speech is acceptable, they slowly start eroding rights to include simply anything that they don't like.

That's why people say that taking away the rights of one group is like taking it away for everyone.