The reason free speech is special is that it can be used to rescuscitate the other rights if they are infringed, at far lower catastriphic risk than, say, freedom to bear arms.