Go tell that to all the minorities around the world who have state granted "freedom of speech" but are suppressed/controlled by mobs in other countries.

It's counter to freedom of speech and it isn't acceptable.

I'd argue that's more a problem with rule of law than anything freedom of speech related.

Freedom of speech is about law. The first ammendment says:

"Congress shall make no law ... abridging the freedom of speech, or of the press; or the right of the people peaceably to assemble, and to petition the Government for a redress of grievances."

Note that that law says nothing about retaining your place in society.

It doesn't have to, because the effect of any law is to define what you can and cannot do in society without being punished, killed or excommunicated by the society, as represented by the government.