Altering the manufactured stone is a deliberate act, and appropriate safety precautions can be mandated. (If people choose not to take those safety precautions, that's on them.)

Asbestos can get into your air through accidental damage to insulation, or in the case of lead, paint chipping or pipes aging can do the trick. This is just something that happens incidentally through the regular lifetime of the material, not because someone decided to do something to it.