How are plastics different than other un-metabolizeable (?) substances (dirt) that organisms have been accidentally ingesting since time immemorial?

Some are probably toxic, but so are naturally occurring heavy metals, etc.