I use PingID for authentication and the manual code entry is only required as a fallback. When everything goes right, you just need to swipe to approve at the lock screen. Also, I'm doubtful there's any cause for concern about malware or physical tampering on iOS.

I do like the idea of being able to plug in to authenticate for ssh on any computer, however...