I genuinely want to understand this so I'm hoping you or someone else can explain.
Yubikey's and the like are a high target item, but probably only for government-level adversaries, in general you have to be pretty well funded to attack these things, plus their attack surface is very very small, and most all of the attack surface is physical (i.e. you have to physically get the yubikey device to attempt any hacks on it). These devices generally only have a USB port, so to get at them remotely you have to first get remote access to the device(computer) they are plugged into, making it even harder of a target.
That's the general overview. Specifically, SMS received tokens are not secure (because of the SMS part not being secure). NIST recently declared SMS based 2FA to be a bad idea, for instance.
https://www.wired.com/2016/06/hey-stop-using-texts-two-facto...
Other comments expalain the vulnerbility of phones, but Linode's Time-based One-time Password implementation (tied to a phone app) may have been hacked server-side.
http://www.securityweek.com/how-attackers-likely-bypassed-li...
The only thing TOTP protects against is somebody collecting logins and using them later. Today good phishers do attacks JIT, so TOTP is pretty useless for that attack vector (and that is the most importent one).
Your Yubikey can pretty much be connected to your computer all the time. So usability is actually better.
Plus there is of course the problem of smartphone security.
If someone can remotely root your phone, they can remotely retrieve your shared secret, and thus generate your authentication tokens.