Indeed, that's a power you'd have to be comfortable having in the hands of your opponents. Because at some point it will be and then they'll be the ones defining what speech is restricted.