The thing is that freedom of speech isn't all that important in the long run. It's important, yes, but speech is the first, tiniest baby step in a very long process towards change. To focus on it to the exclusion of all else is to no different than silencing yourself.

Indeed, having speech and naught else is like a vent that lets off steam, making change and better lives harder to come by. It is the fool's prerogative to speak his mind, but at the end of the day, the king is still a king, and the fool is still a fool.