The first amendment also protects freedom of association. You don't have a right for people to hear you, but the government doesn't have an unlimited power to restrict your ability to actually reach people.

If government prevents people hearing you speak, that's a pretty serious curb on freedom of association.