No that's wrong, authy backs up your TOTP seed to their server and will give them out to anyone who gains access to your authy account and can verify that they can receive sms messages sent to the number associated with the authy account. (Though they are encrypted and authy does not store the encryption keys).

That's a vulnerability of Authy's service, not TOTP.

I have some of my more important services tied to an older Yubikey. The only attack vectors for it would be on the server-side and the physical key.