Well, personally, if I attended or worked at MIT, I would not want Nazis on campus and would accept banning them.

Great, you're free to not want them, but everyone has to justify their actions, so an action like banning a group still requires justification. If that justification is sound and based on accepted ethical principles, it should arguably become an ethical rule. I personally haven't seen an argument that satisfies the requirements.

I don't think I've ever met a human who was incapable of justifying their actions.

They're all capable of attempting to justify their actions, but rarely is that justification sound. "Justification" here means a deductive argument from principles, sort of like what you'd find in a court of law based on except based on ethical rather than legal principles.