Most particles can be removed by your immune system. Things like asbestos are bad specifically because the kinds of particles asbestos becomes are very difficult to phagacytose (and remove).