I think that maybe this falls under the paradox of tolerance (https://en.wikipedia.org/wiki/Paradox_of_tolerance)

I think that you are right. It seems that a tolerant society attracts intolerant (both far left and far right) elements that want to restrict the freedom of speech. It is similar to the reason that GPL is prefered by some - by restricting some freedoms you defend the others. Similarly I think that it would make sense to restrict the freedom of speech of these that want to eliminate it.