However in the absence of U2F, you can still use Yubico OTP (HOTP) or you can use your yubikey together with your smartphone for TOTP. I think its a bad idea to use your smartphone as a 2Factor. However, if you combine your smartphone with your yubikey (download "Yubikey Authenticator") you can use your smartphone simply as a time provider.
This has the added benefit that if you lose your phone, you can use any other phone/table to do the authentication. Google Authenticator does not support that feature, and Authy Cloud-based solutions are not for everybody.
I use my Yubikey in all 3 different ways. I use Yubykey+Smartphone if I have no other options. Yubykey as U2F were I can (Dropbox, Github, Google) and Yubikey OTP (HOTP) for some services that support that instead of U2F.
Additionally you can use the Yubikey to decrypt your harddrive with HMAC or Static Password.
I really hope OpenSSH (and everybody else) will support U2F soon, its so much better then bothering with the smartphone and typing stuff into your computer.
The simplest way is just to do a long static password and combine it with your normal password. So you type '1234' and then hit the yubikey button. The problem with this is that temporary access to your yubikey allows somebody to steal your static password.
The better way is to use the Challange-Response mode that send your password, the Yubikey sends back a hash (HMAC-SHA1).
Their is very new way that you can use that uses time based authenication, but this was just presented at 32C3, see: Beyond Anti Evil Maid [32c3]
The problem also is that the smartphone can be hacked and opens a huge attack vector that you don't have with other systems.
Also, typing in stuff by hand is a general bother.
There is also the matter of control. At the moment there are only a handful of smartphones that run an OS that is not under the control of Apple, Microsoft, or Google (Ubuntu smartphones come to mind as an alternative). On the other hand, on a normal computer I can run a full free software stack. To me this seems odd. If I own a piece of hardware, I want to run an OS and applications I can legally modify and tinker with, especially if I use it as a security device. By using tools such as Google Authenticator I get sucked into this walled garden.
The behaviour of hardware tokens such as Yubico's U2F keys on the other hand is completely verifiable — mostly because they are quite simple compared to a smartphone.
(Aside from these personal views on software, I do have high security requirements as well for some services, where a dedicated hardware token is preferable, as you point out.)
From a software freedom perspective, hardware tokens/smart cards are no better than smartphones.
I agree that the ideal situation would be a device that is completely under the control of the user, but compared to IOS or Android, a hardware token that only does what it is supposed to do (in the case of the U2F tokens; register and sign authentication requests), and cannot be modified by the manufacturer after sale is very much preferable.
Alternatively, you could go for a dedicated hardware token such as Yubico's FIDO U2F [1] keys [2]. Hardware tokens that use the fledgling FIDO U2F standard can also be used with PAM and SSH as well [3].
Two U2F keys (one backup) will cost around $40, and can be used with a growing number of webservices, including GitHub, DropBox, and GMail as well. The small form factor means you can put the key on your (physical, real world) keychain.
1: https://fidoalliance.org/specifications/overview/
2: https://www.yubico.com/products/yubikey-hardware/fido-u2f-se...
3: https://developers.yubico.com/pam-u2f/