At the core of the first amendment is the idea that people should not be punished for criticizing their government. I think that idea is worth preserving. But the idea that people are free to say anything they choose, in any context, regardless of its factual status, and also that their permission to do so is limited only by the resources they can muster to promulgate their speech, is an unwarranted extension of that concept.
The cold hard reality is that no matter how much you trust the people in the government today, eventually they will be replaced by people you consider to be the scum of the earth. And when that day comes, you will curse the day you allowed the government to punish speech, because you'll see speech you consider perfectly justified become illegal.
There is no way of listing rights on paper that can protect you if truly evil people get into power. But there are ways of listing rights on paper that can allow good people who believe in those rights to defend them in ways that involve preventing evil people from getting into power. Free speech is not a magic bullet in either direction.