The problem is that our culture doesn't really agree on whether it's the responsibility of the speaker to avoid offense or the listener to "suck it up" or whether it varies based on the relative identities of the speaker and listener in a hierarchy of identities. I'm fine with expecting courteous speakers and charitable listeners, but running it through an identity matrix is inherently racist (and/or sexist, and/or etc).