Statistically, the DIYers need not really think about it, either. If you work with it at most a few days of your life. you can be the unlucky guy who breathes in a lot of asbestos dust and dies, but chances are you won't (if you don't go berserk when demolishing a building, and are smart enough to wear a mouth/nose cap when working inside in dust)
Professionals in the industry, on the other hand, may work way more carefully, but still run a much larger risk of adverse effects due to their longer exposure time. That's another reason the men work in bunny suits.
With care, most of this can be removed with little risk. However, as you point out, caution is needed and most DIYers may not think about things. The other extreme is calling in the men in bunny suits that will run multiple thousands for even small issues due to permits, precautions, and disposal.