Those limits are - by definition - set by the powerful and at their most effective on the powerless, because the ability to control what kinds of speech are allowed in a way that affects large numbers of people is in itself a huge form of power. If anyone claims otherwise, you should take a good look at their motivations.