>you need another backup option before you're allowed to remove the SMS option.

This was the answer. Google prompt isn't allowed with hardware tokens. Backup codes evidently don't count. So the only way is to set up Google Authenticator on a phone. Authenticator from f-droid works. After I set up Authenticator, I no longer got the "Something went wrong. Try again" toast when trying to delete the sms number.

Edit: Just realized what Yubico Authenticator is for :)