>you need another backup option before you're allowed to remove the SMS option.
This was the answer. Google prompt isn't allowed with hardware tokens. Backup codes evidently don't count. So the only way is to set up Google Authenticator on a phone. Authenticator from f-droid works. After I set up Authenticator, I no longer got the "Something went wrong. Try again" toast when trying to delete the sms number.
Edit: Just realized what Yubico Authenticator is for :)
You might need to remove it as an account recovery number as well. Those can effectively downgrade your login to one factor.