Libertarianism works better without libertarians. It works better with collectivists than individualists. It works better with people who care about other people's freedoms than with those who care about their own freedoms.

Freedom of speech as a negative liberty means that the government does not criminalize speech. Some people would like to extend that to other powerful entities banning / limiting speech, but then the government would be suppressing those entities' other liberties.

The freedom of speech people care about is a positive liberty. It's about being entitled to have a platform for your speech. Maybe even a platform with an audience, or a platform you feel safe to use. In order for your opponents to have freedom of speech – a platform – your liberties must be suppressed. Either by an authority by force, or by yourself voluntarily.

Ultimately, freedom of speech is about you being obligated not to confiscate the megaphone.