Not really, it becomes very difficult to remove a phone number from an account and anyone who gains access can use various account recovery mechanisms which rely heavily on access to that number for authentication.