It's clear why this happens, having difficulties is viewed as undesirable, most people would choose not to have them if they could (I'm aware some people with disabilities have felt it made them who they are and wouldn't change it, but they aren't the people to start using terms in a pejorative manner). As long as there are real negative attitudes against black people and real negative effects on black people, this is likely to be the same for terms used there also.
I feel the only reason person centered language like "people of color" and "person with disabilities" has lasted so long as the acceptable phrasing is simply because the more wordy phrases are harder to turn into a playground insult/drunk jeering/etc.
I don't really have any suggestions about how this could be handled differently, just pointing out this aspect that does end up making me talk less, certainly about touchy subjects, because I'm afraid of being misinterpreted and seen in a bad light.
Edit: It's also especially bad because English isn't my native language so I might use some construct that seems "natural" to me but insulting to a native speaker
Here we have a too-common case of virtue chasers on the "same side" attacking each other because they have slightly different useless solutions to a problem. (The way you to show more respect to different people with differences is to show more respect in actions, not to play word games that the vast majority of affected people with effects don't care about.)
If anyone cared what the group being described thought about the terminology, no one would ever have tried to say "Latinx".
Like I said above, pure fashion.
I had a look, the wikipedia page[1] indicates at least in the US that the American Medical Association, United Spinal Association and various federal and state bodies recommend it as best practice, but also points out various groups for deaf and blind people disagree.
[1]: https://en.wikipedia.org/wiki/People-first_language#Rational...
1. https://www.disabled-world.com/definitions/disability-disabl...
See also the "Spastics Society", "National Association for the Advancement of Colored People", modern use of the n-word among black people, the use of "Negro" in "I have a Dream". Plenty of cases where a word in use by the group themselves becomes considered the impolite word.
[1]: https://www.youtube.com/watch?v=kh88fVP2FWQ
[2]: https://en.wikipedia.org/wiki/Native_American_name_controver...
Approximately 97% of "Latinx" people do not identify as "Latinx".
It's different in that it's a different phrase. It's better because it's different; terminology in this area is driven purely by fashion.
https://www.gocomics.com/bloomcounty/1988/08/28