Yes, but that's more of a general argument against all SSO - besides, does your concern hold even for things like Sign In with Google / Apple? I assume losing access to iCloud / Gmail (if you've opted into these ecosystems) is an existential threat anyway, and SSO is the least of your worries at that point, and the question still stands.