Being mugged isn't "user error", and the good part about YubiKeys is that the consequences of being mugged are much more intuitive (it's just like I had my keys stolen).

The "I left my keys in my coat" problem you have seems like it has a pretty obvious solution: don't keep your YubiKey on your key chain. Do you have a wallet or cell phone that you carry with you at all times? Put it in the wallet or cell phone case.