A family member used to work for a company that produced asbestos cement products. I can still remember touring the factory as a child: the grinding machines shaping the pipe couplings, standing on top of the dust collector as it was being emptied, the sealed room at the beginning of the production line into which the raw asbestos was being blown. (I only looked through the glass door of that one, though I gather it wasn't unknown for people to be locked in there for a joke, whilst workmates enjoyed the "snow show".)

Yet, for that, some people who worked at the company are still healthy (and being regularly monitored), whilst others are dead from mesothelioma. It's as if some people are susceptible, whilst others aren't, and size of dose isn't the overriding factor.

I don't think it's overblown, in that people do die (unpleasantly) from minimal exposure to asbestos. Perhaps that's why asbestos is treated so gingerly: to some people it is as dangerous as nuclear waste and we don't know who those people are?

--- Edit: final question mark.