I disagree. Language is very often used as a way to include and exclude people in and from various social groups. You're right in that there's never any direct harm in the sound of the words themselves. But you can say the same thing about verbally threatening physical harm (or even psychological harm) to someone, and would be very wrong in your assumption.