I mostly use the same "non-contact lenses" as you but have also used contact lenses for 35 years and not needed to have an eye removed.
Deliberately putting things in your eye does have one advantage, you can easily fish out any bits of grit that get in there by accident.
Ask me on my death bed to be sure - but I'm pretty sure that the risk of eye removal from my non-contact lenses is pretty low. I much prefer it to the creaky old technology of actually putting things in your eye. Progress!
(I used to have some fashioned from pure glass, and they were pretty decent - much more resistant to scratches - but obviously glass comes with a risk of breakage and therefore shards of glass in your eyes. I have to admit that such a thing never actually happened while I had them, though.)