So when a word that starts out as a neutral term to describe some condition or collection of people gets adopted over time by the general public as a slur or derogatory term, and that's how it is used 99.9% of the time, we should assume when we hear it out of context it is the 0.1% case?

Also for many of these words most the general public has complete forgotten the non-slur/non-derogatory meaning (or has never known it). No amount of tone will fix that.

Imagine an unmarried couple with a child who gets a copy of the child's medical records from their pediatrician and see the child described as a "retarded bastard". The chances that they will know that "retarded" was once a medical term for someone who is behind on mental development and that "bastard" means a person born out of wedlock are pretty low. At best they are going to be greatly puzzled by the doctor using those words.