We can agree to disagree here. In the end, I would rather converse with someone who errs of the side of not wanting to "offend" me, but can still freely say whatever they want vs. someone who speaks with no regard to how their words might affect me and makes no apologies for it.
The outcome is the same, but to me, one person is thoughtful and the other person is a jerk. I have never thought that someone trying to be thoughtful with their words was patronizing me, but that's only my experience.
I have the opposite visceral response to this sort of thing. If someone refers to me either singly or as a part of a collective such as "persons having a mental impairment" I immediately know I'm dealing with the sort of bastard that no human connection can be formed with.
The language betrays the intent, and the intent is to proclaim a shibboleth for the purposes of in-group/out-group reinforcement.
Because it's patronizing. You can use normal language without intending to offend someone, but if you actively censor your own language in their presence, you're removing their agency.