Just speculating here, but doesn't it have something to do with accumulation in the body? Your body obviously can't accumulate radiations. It also does process and eliminate the like of ethanol and many other substances. That's not the case, I believe, for heavy metals like mercury, lead or arsenic, that are eliminated much more slowly (if at all?).

That's why you can enjoy lychees by eating a few everyday if you fancy it, and will only poison yourself if you eat a lot at once. But you can poison someone by exposing them to a little bit of arsenic everyday (at least in the movies, not sure how true it is...)

I think tar from smoke accumulates, and there is a lot of it in second-hand smoke.

Obviously not a doctor, so I'm probably completely wrong...