If you have a Yubikey NEO and a phone with NFC, you can store the TOTP secret on the Yubikey instead of the phone, and use the phone purely as a screen with a time source by opening Yubico Authenticator app and touching the Yubikey to it to generate codes.