It's even easier for Github and Google Mail. For web services, the right stack is:
* Hardware U2F token
* Backup software TOTP (Duo or Google Authenticator or whatever)
* Backup printed (or saved on offline USB key) passcodes
* Disabled SMS.
Unlike SMS, which is devastating to security even as a fallback, having a software TOTP option is basically fine; most of what U2F buys you is unphishability. This leaves you with two levels of backup, one of which is reasonably secure indefinitely.
But if I'm, say, protecting my GitHub account against Russian mafia hackers, that still seems perfectly fine?
You might need to remove it as an account recovery number as well. Those can effectively downgrade your login to one factor.
This was the answer. Google prompt isn't allowed with hardware tokens. Backup codes evidently don't count. So the only way is to set up Google Authenticator on a phone. Authenticator from f-droid works. After I set up Authenticator, I no longer got the "Something went wrong. Try again" toast when trying to delete the sms number.
Edit: Just realized what Yubico Authenticator is for :)
You have to have TOTP and Backup keys. Maybe this is a recent change.
I've been SSH w/ yubikey key only for about 6 months now and haven't had any issues with it. I regularly move between multiple computers. Once I set it up on one computer I've never had it take more than 15 minutes to get up and running on any computer I've need it on (window or linux).
The only thing I'm really missing is the ability to log into my server from my phone. There was some talk of getting ConnectBot and Open-Keychain talking to each other to get this working but it appears to be stalled.
U2F (Google, Dropbox, Facebook, Github, Bitbucket), TOTP (Slack and everybody else that does not support U2F), Yubikey OTP (LastPass), static password for luks decryption (additionally to normal password), GPG Smartcard
The only feature Im not yet using is the PIV SSH stuff.
I also just like hitting the button and printing out OTPs when Im boarded.