Yes but your backup is encrypted by a password. So even if someone steals your number for long enough to go through recovery, they still need to be able to decrypt the backup.

>this password is not stored anywhere on Authy's servers! If you forget the password and none of your devices are synched, your tokens are lost and you will need to delete them and start over

https://support.authy.com/hc/en-us/articles/115001750008-Bac...