I get master/slave being potentially offensive if you apply it to history, but blacklist/whitelist etymology was never related to race at all. Blacklist seems to come from a more religious perspective with people without good souls being empty or black inside.

If people actually find it offensive then by all means change it. However I wonder if people read too much into thing sometimes.