That's just not true. A secure enclave could, for example, wipe old keys when adding new keys; a physical attacker might succeed in adding his new key, but my data would be secure. That data could include whatever mechanism the phone uses to authenticate itself to me.

I don't trust Apple to be good from now until I replace my phone. I don't trust Google to be good from now until I replace my phone. It's my phone, not Apple's and not Google's.