This is true for Google, Facebook, and Twitter logins as well. It's a valid criticism for using any third party authentication provider (which I personally avoid and will be avoiding Apple's).

The login provider doesn't store the service account itself, so it's possible to "eject" your account and use some other sign in mechanism as long as you have control over the email address.