A passphrase doesn't really turn a ssh-key into 2fa. It's still really only "something you have (access to)": the plaintext of the private half of the key. Granted, you can use ssh-keys with tokens -- my recommendation is to use encrypted ssh-keys paired with a recent[1] openssh server that allows stacking OTP on top, for 2fa with the RequiredAuthentication-directive (well, OTP is generally code for: can demonstrate access to OTP secret) -- but at least it is rather easy to store the two on separate devices (say a phone, and on the filesystem of your workstation/laptop). Not really maximum security, but quite a lot better than traditional passwords.
Until you store both the key and the authenticator on your phone, in order to use ssh from your phone. At which point you end up with "rather convoluted 1fa" -- which might still be marginally better than traditional passwords (along with a warm fuzzy feeling of false security :-).
Other than that I've been looking into using openssh certificates with openssh, but it still looks like it's rather complicated -- and for a single-/few-user setup it's questionable if the expiration etc really adds much actual security.
Maybe the best would be to apply some of the kerberos design lessons to openssh with certificates, issuing short-lived certs much like kerberos ticket-granting tickets (so have an easy way to get a user certificate that's valid for 8 hours or so...). Not sure the added complexity would be worth it, though...
[1] https://blog.flameeyes.eu/2013/03/openssh-6-2-adds-support-f...
Assuming the ssh key is used with a pass-phrase, it's hard to imagine any real scenario where the key is compromised, but not the password.
I suppose an unencrypted key might be lifted from compromised backups (but not passwords, assuming secure passwords, and secure hashing -- defeating even off-line brute-force). However, an encrypted key should be at least as secure (again assuming a "secure" pass-phrase).
OTP (especially on a separate device, such as a phone) has the benefit of not being compromised (permanently) by a keyboard-logger/root access on the terminal. On the other hand it is probably vulnerable to an attack on the device.
I also find it comparatively easy to manage one OTP secret per account, vs passwords, where they tend to end up becoming opaque tokens (ie: managed by a password manager -- and by extension, no longer "something you know", but rather "something you have"...).
One of the worst part of passwords, is that even if it is stored securely (salted, hashed, stretched) -- it is essentially a shared secret: the server has the opportunity to log it on every login/use. While a secret key can remain secret while you use it (as long as your terminal is secure).
I wonder if there's been any work towards asymmetric-key (T)OTP? I suppose it shouldn't be to hard, but I'm not aware of any established systems for it. Should be enough to use ECDH to derive a symmetric key, and then use that (or a derivative) with traditional TOTP.
Just found this[1] -- not sure if it's really relevant, though. Strange to find something like this with such a recent filing date on it, though.
I think key + pass + OTP is the best of all worlds, except convenience. But in my opinion, it should be a pain to get into your servers, especially if they hold sensitive data. I am especially a big fan of hardware tokens like yubikeys; the best part is, you know when you lose them and can rotate keys. Even with Google Authenticator, you are not quite sure if the keys can be lifted from your phone.
[Edit: Removed redundant comments.]
For something where it's more "lots of users need to connect to lots of boxes regularly", I'd prefer kerberos or LDAP/AD (to manage keys).
There's also a good argument against ssh keys anyway -- they don't have expiration dates, don't necessarily provide 2fa, don't have good secure storage on client devices, etc.
(In the long run I'd probably go with a client-device managed credential, some kind of client-device-specific biometric or other auth through a secure path, and some scheme to ensure users don't authenticate via a tampered client device. It's a hard problem in a mobile/laptop environment.)