Demolition, sanding drywall compound all is bad without PPE and Hepa filters.
You can buy a cheap yet very comfortable 5000 or 7000 series respirator from 3M for about $20 and the "pink pancake" P100 filters are another $12. If you're a hobbyist those ought to last you a year, maybe even 5 depending on what you're cutting. I do metalwork as a hobby and I can definitely feel it in my lungs (and taste it) the next day if the garage door is shut and I forgot to wear the respirator.
Effects depend on the particle geometry.
But as a general rule of thumb: if it's biologically inert and tough enough to stick around for awhile... it'll just stay in your lungs. E.g. fiberglass, cement, rock, paint.
I wear a respirator a LOT more these days than I did when I was younger.
Yeah, but it doesn’t kill you within a few years if you don’t, right?