A family member used to work for a company that produced asbestos cement products. I can still remember touring the factory as a child: the grinding machines shaping the pipe couplings, standing on top of the dust collector as it was being emptied, the sealed room at the beginning of the production line into which the raw asbestos was being blown. (I only looked through the glass door of that one, though I gather it wasn't unknown for people to be locked in there for a joke, whilst workmates enjoyed the "snow show".)
Yet, for that, some people who worked at the company are still healthy (and being regularly monitored), whilst others are dead from mesothelioma. It's as if some people are susceptible, whilst others aren't, and size of dose isn't the overriding factor.
I don't think it's overblown, in that people do die (unpleasantly) from minimal exposure to asbestos. Perhaps that's why asbestos is treated so gingerly: to some people it is as dangerous as nuclear waste and we don't know who those people are?
--- Edit: final question mark.
I remember when I was a carpenter apprentice and asked for personal safety equipment and then being told that "wood dust never harmed anyone" or "I never use hearing protection and my hearing is just fine". I have encountered this attitude in many places, especially in small companies where not following safety rules can be seen directly in the profits. But you also see it in big projects, where the main contractor hires sub-contractors that hires sub-sub-contractors and no one has an overall responsibility.
The other problem is that it's easy to accidently make dust out of otherwise 'safe' asbestos, at which point it's too late. For example, hit and break a wall for whatever reason (put a hole in an internal wall, fallen branch hitting a roof), or drop some sort of heat-resistant cooking device that breaks when it hits the floor. It's one of those things where even if you do know of the risks, the possiblity of it being accidently realised is significant.
I fully acknowledge that some of the material properties of asbestos are truly amazing - its heat resistence is incredible and we'd be well served finding a safe equivalent - but on balance it is simply not enough to justify its use in applications where the layman can be exposed to it.
For example, many houses in the Bay Area have asbestos in it, so it's not like having it in your house is going to kill you. You need to deal with it intelligently, but just having asbestos in your home isn't going to kill you. And even then, unless you're breathing it in all the time, you're not automatically going to get lung cancer, just like if you get a CT scan with a radiation exposure equivalent to hundreds of X-rays, you're not automatically going to get cancer. Proper education is important. Sure, lots of people tragically got lung cancer due to heavy exposure, but some exposure won't guarantee anything bad, but it certainly feels like you will based on how people talk about it.
With care, most of this can be removed with little risk. However, as you point out, caution is needed and most DIYers may not think about things. The other extreme is calling in the men in bunny suits that will run multiple thousands for even small issues due to permits, precautions, and disposal.
Professionals in the industry, on the other hand, may work way more carefully, but still run a much larger risk of adverse effects due to their longer exposure time. That's another reason the men work in bunny suits.
For most members of the public, our interaction with Asbestos will be from it's use as a building material for roofs, insulation or pipes. The risk only increases if it is heavily disturbed, usually due to building work or demolition.
Asbestos and all commercial forms of asbestos are known to be human carcinogens based on sufficient evidence of carcinogenicity in humans.
You can read the rest if you want more info:
http://web.archive.org/web/20110429155522/http://ntp.niehs.n...
The government and real estate agents treat it as if it's nuclear waste, but being around asbestos isn't guaranteed to give you lung cancer. It's only when you create a dust from it by grinding it, and if it gets airborne. There are main mitigation techniques that can control asbestos pretty easily. Having it in your house isn't terrible as long as you deal with it properly.