I mean, you do realize that 99% of black people have no idea where "grandfathered in" comes from, right? You do realize that going into etymology to find offense is something only snobby linguistics cunts that you would find at elite universities would do, right?
In society we have come to a collective understanding that “I’m not racist; some of my best friends are black” is an invalid dismissal of a person’s potential for racism.
Somehow, however, we have also decided that if someone does not have a sufficient quantity nor quality of relationships with black people, then that is a strong signal of racism.
Damned if you do. Damned if you don’t.
Labeling a word as harmful actually harms more, since most new generations are so removed from the word's origin that it carries none of the supposed harm
Are they the only ones who tell you, a fellow white man, or are they the only ones who are offended? I strongly suspect (this is not meant to be insulting) that you may not have a relationship with the people who this language impacts. It takes an emotional toll to confide with someone that something is upsetting to hear, even with coworkers or casual friends, especially if you don't think they'd understand.
Also, language has the opportunity to shape and influence how we structure society, even unintentionally. Thats why there's a lot of emphasis on teaching children about what words are appropriate. Even if naming your git branch "master" won't realistically impact racism and the history of slavery, describing some bad event in your day a word that has an origin in describing people can lead to negative associations.