Government is just one form of a society's self regulation.

If free speech is a "right" is there a qualitative difference between society restricting it via one mechanism but not another? Does it matter that its your neighbors beating you to shut you up instead of the cops? If so, why?