If you have a Yubikey NEO and a phone with NFC, you can store the TOTP secret on the Yubikey instead of the phone, and use the phone purely as a screen with a time source by opening Yubico Authenticator app and touching the Yubikey to it to generate codes.

For Fastmail I plug it into the USB port as usual (using an OTG cable). Last I checked that was still the only way to do it specifically for Fastmail, since they do not support NFC. For Lastpass though, using the Yubikey through NFC works great.