Firstly, "things that shouldn't be said" include matters of etiquette and not just ethics, so just be clear about the line between being a good/ethical person vs. being a polite person.

Secondly, your personal ethics are absolutely relevant because ethics advances by persuasion in rational debate. Everyone's views on such matters are relevant, if not persuasive.

Well, personally, if I attended or worked at MIT, I would not want Nazis on campus and would accept banning them.

Great, you're free to not want them, but everyone has to justify their actions, so an action like banning a group still requires justification. If that justification is sound and based on accepted ethical principles, it should arguably become an ethical rule. I personally haven't seen an argument that satisfies the requirements.

I don't think I've ever met a human who was incapable of justifying their actions.

They're all capable of attempting to justify their actions, but rarely is that justification sound. "Justification" here means a deductive argument from principles, sort of like what you'd find in a court of law based on except based on ethical rather than legal principles.