I think it is an oversimplification to say that free speech is a general ethical ideal. As with many things, the Constitution is attempting to strike a balance. The check on restriction of speech is the prohibition of the government to restrict speech, but there is an opposite check as well: private entities are permitted to restrict speech. In the most extreme case, this means people can determine for themselves what is discussed in their own homes. There is a lot of tricky gray area between private homes and the public square.