The term “plastics" contains a big bag of substances that have very different chemical structure and properties. We don't know what the long-term effects of many of these substances are. Stuff like Dioxins and friends are obviously bad, and still took a significant amount of time to root out. Plastics are everywhere now, so in case it turns out that more substances are bad, we are in deep shit.

Edit: regarding non-organics, it took a long time to realize that inhaling fine dust causes long-term health problems, and quite longer until dust filters became required and commonplace. We are still not done regarding fine dust emitted by traffic.