There's a difference between no known mechanism (and the effect being completely swamped by the environment) versus actually having a contaminant persist in the body, which historically is usually the precursor of larger effects.

I.e. asbestos is harmless, and would be harmless if didn't remain in the lungs, and we just breathed it back out.

Hell, you can handle most radioactive isotopes quite safely provided they don't enter the body in a persistent way.