I think that might be the most important time to listen to them. If merely hearing an opinion threatens your model, that’s probably a warning sign indicating that your own ideas are fragile and unsustainable, and they need to be exposed to ideas that challenge them. Engage with painful ideas, break your own models down and reformulate them into something more robust. I think the unwillingness to do this is what leads to the problems you mention.
I’m reminded of “The Coddling of the American Mind”
This doesn't strike me as a universal truth. It's good as a general principle to test your ideas against ones that challenge them and make sure they're as robust as possible -- but if that "challenging idea" is "your group has no right to exist," then it's not reasonable to argue "if merely hearing that you have no right to exist threatens your model, that's a warning sign that your own ideas are fragile and unsustainable," is it?
My point was more like, if person X really does feel that way, then sure they can retract themselves from discussion. But they should do so knowing it’s their own failing/weakness and not blame it on the other person being “toxic”.
Over time I've come to realize that usually there's a legitimate reason why someone believes what they do. Often they are either optimizing for different things, or view the matter differently.
If listening to someone's opinion is actively causing emotional harm, then sure, don't listen to them. But I worry very much about the rise in offense-taking. Perhaps it's an illusion but I've felt it's become incredibly difficult to talk to people with different [political, etc] opinions than my own: not because of me, but because of their attitude towards dissent.