Great, you're free to not want them, but everyone has to justify their actions, so an action like banning a group still requires justification. If that justification is sound and based on accepted ethical principles, it should arguably become an ethical rule. I personally haven't seen an argument that satisfies the requirements.