It's worth noting that in this case we're only talking about keys that are useful to someone in physical possession of a phone. So as long as I retain physical possession of my phone no one can get at it, keys or not.

That's still pretty safe.