Is remote desktop compromise less likely than remote phone compromise? Phones tend to sandbox each app, so a single malicious or compromised app cannot take over your phone. Desktops on the other hand allow any app to do essentially anything on your computer. Accidentally open a malicious email attachment and you're done for. Generally I consider machine/phone compromise outside of my threat model.
Is security key theft less likely than phone theft? Also, phones generally require a password or fingerprint or face to unlock, while security keys generally do not (Yubico announced a fingerprint-protected one 18 days ago, but it's not available yet).
I bring my phone with me in lots of circumstances where I don't bring my security key, so yeah I think it's more likely to be lost/stolen.
Also, TOTP has a MITM problem that U2F/Webauthn doesn't have.
I almost never use Twitter on my phone, and if I switched Twitter to depend on my Yubikey, I would stop using Twitter on my phone completely.