How do you use a yubikey on your phone? Or did you mean in lieu of totp on your phone?
If you have a Yubikey NEO and a phone with NFC, you can store the TOTP secret on the Yubikey instead of the phone, and use the phone purely as a screen with a time source by opening Yubico Authenticator app and touching the Yubikey to it to generate codes.
For Fastmail I plug it into the USB port as usual (using an OTG cable). Last I checked that was still the only way to do it specifically for Fastmail, since they do not support NFC. For Lastpass though, using the Yubikey through NFC works great.
So yeah, if I have my phone, I have a 2FA option already without SMS. I normally use the Yubikey because it's super easy.