Fiberglass, dust, and so forth can be expelled by the body and don't represent nearly the same level of harm as asbestos. The material's mechanical and chemical properties make a huge difference in how dangerous they are. Asbestos is chemically robust and mechanically fragile in a way that makes it more dangerous and sticky over time.
A nanotube that damages a few cells, then gets metabolized or oxidizes, and then expelled, is far different from a slowly exploding needle bomb that will reside in your body for decades, endlessly killing the cells it contacts, resulting in infections, inflammation, cancer, and sometimes even dead septic chunks of tissue.
Asbestos is, on balance, a terrible, horrible thing, and the harm it does can't be justified by the potential for good uses. Fiberglass insulation or carbon nanotubes aren't good for your lungs, but the dangers they pose can be reasonably considered against their benefits. these materials present a very different scale and magnitude of harm, especially over time.
Synthetic vitreous fibers deposited in the gas exchange area of the lungs also slowly dissolve in lung fluid. Fibers that are partially dissolved in lung fluid are more easily broken into shorter fibers. Shorter fibers are more easily engulfed by macrophages and removed from the lung than long fibers.").
We also have been unable to find clear evidence of health harms in longitudinal studies of fiberglass manufacturing workers: https://www.atsdr.cdc.gov/ToxProfiles/tp161-c2.pdf ("Studies of workers predominantly involved in the manufacture of fibrous glass, rock wools, or slag wools have focused on the prevalence of respiratory symptoms through the administration of questionnaires, pulmonary function testing, and chest x-ray examinations. In general, these studies reported no consistent evidence for increased prevalence of adverse respiratory symptoms, abnormal pulmonary functions, or chest x-ray abnormalities; however, one study reported altered pulmonary function (decreased forced expiratory volume in 1 second) in a group of Danish insulation workers compared with a group of bus drivers.").