For example, many houses in the Bay Area have asbestos in it, so it's not like having it in your house is going to kill you. You need to deal with it intelligently, but just having asbestos in your home isn't going to kill you. And even then, unless you're breathing it in all the time, you're not automatically going to get lung cancer, just like if you get a CT scan with a radiation exposure equivalent to hundreds of X-rays, you're not automatically going to get cancer. Proper education is important. Sure, lots of people tragically got lung cancer due to heavy exposure, but some exposure won't guarantee anything bad, but it certainly feels like you will based on how people talk about it.
With care, most of this can be removed with little risk. However, as you point out, caution is needed and most DIYers may not think about things. The other extreme is calling in the men in bunny suits that will run multiple thousands for even small issues due to permits, precautions, and disposal.
Professionals in the industry, on the other hand, may work way more carefully, but still run a much larger risk of adverse effects due to their longer exposure time. That's another reason the men work in bunny suits.
The other problem is that it's easy to accidently make dust out of otherwise 'safe' asbestos, at which point it's too late. For example, hit and break a wall for whatever reason (put a hole in an internal wall, fallen branch hitting a roof), or drop some sort of heat-resistant cooking device that breaks when it hits the floor. It's one of those things where even if you do know of the risks, the possiblity of it being accidently realised is significant.
I fully acknowledge that some of the material properties of asbestos are truly amazing - its heat resistence is incredible and we'd be well served finding a safe equivalent - but on balance it is simply not enough to justify its use in applications where the layman can be exposed to it.