True- I don't think they are likely to specifically exhibit the drug like toxicity that some other plastics are responsible for, but you are right that doesn't rule out other potential issues from a small, hydrophobic, physical object.

I'm not 100% sure HDPE is safe- and if not it might depend on how it is used, but I am less concerned about it than a lot of other plastics.