For SMS auth, you can get the cell company to re-issue a sim if you lose your phone. This is a feature and a bug, thanks to social engineering attacks. Also, the SMS transport layer isn't secure.

With TOTP, you can put the TOTP code in your password manager with all your other passwords, which is probably what I will do it it catches on. It's effectively one factor at that point. (Do you know my password manager's password?)