I gave you an upvote because you’re asking a reasonable question. One answer is that snippets of organic compounds are often carcinogenic and/or bioactive, so they’re scarier than (say) metal filings that just dissolve in short order (try a magnet on “iron fortified” cereal! Turns out iron’s fairly bioavailabile when combined with vitamin C.)