Off-topic: Has anyone ever managed to get PAM/Google Authenticator working with RADIUS? I spent a while messing about with this last year, and never got it working exactly as I had hoped. I'm no longer working with RADIUS, but this post reminded me I never finished scratching that geeky itch.
http://it.isevil.org/blog/2011/11/13/authentication-service-...
Code: https://github.com/bithive/example-totp-vault
For FreeRADIUS we use rlm_perl to define our own authenticate() method; it just calls the web service to validate the codes.
http://code.google.com/p/google-authenticator/source/browse/...
If you lose your phone and your scratch codes, you've only lost access via SSH. So it's an inconvenience, but one you can overcome with the right setup.
Yes, if you lose either factor, you can't access the server. This is why with Google Authenticator you also get a one time pad with emergency codes. However, I don't know how well this would work with two-factor SSH... you'd need a separate one time pad for each server. And I'm not sure how the Google PAM module handles emergency codes.
https://code.google.com/p/google-authenticator/