I think you're describing the right to free association. But when free association is dominated by mob mentality it becomes a minefield for dissenting voices.