The only issue with this sort of setup is when you lose your phone and need to set up a new device. Or if you for some reason need to authenticate to a non-personal device. Smart cards have traditionally been used in this way but they haven't really been updated for the mobile world.

I could see using a yubikey stored securely in your house for just this reason and relying on the secure element in your phone for everything else. If course if you have two phones, or a computer with a secure element (like the new MacBook Pro) you could just use your other device to authenticate.