It doesn't prevent unauthorized use, it just makes it more difficult to attack with brute force. Once someone gets your private key, you should always assume it has been compromised, and replace it immediately. It's only a matter of time and resources before a determined thief has cracked your passphrase.
Edit: not to say I disagree with your recommendation that it's best, just pointing out that compromised private keys should not be taken lightly.
If your key is on a smart card it can not be stolen, even if the machine you're ssh'ing from has been compromised. This has real value and is not "insane"
In your setup, if their machine is compromised, it doesn't matter if they password protect the key, the attacker can key log the key password or replace the ssh executable with one which leaks the key, then ssh into your servers at will.
If the machine you're ssh'ing from has been compromised by a halfway competent attacker, there's no need for them to get the key from the smart card. They can log in to the remote machine by simply piggybacking on to your ssh session (with a compromised ssh client).
Most devs know how to use ssh-agent. Those who don't are really excited to learn that they can increase security without an increase in effort.
Password uniqueness, on the other hand, is something that requires continuous effort. There's an ongoing temptation to reuse the strong password that they've memorized for my server in other locations.
Like most things in security, it's a matter of layering and balance. You layer your security, and you stop at the point where the added effort doesn't justify the added security. I view the progression like this
OK: Strong passwords
Better: SSH-key authentication (much harder to brute force, not susceptible to disclosure through third-party security disclosure)
Best: SSH-key authentication with passphrase (prevents unauthorized use of private key if stolen)
Insane: SSH-key stored on external drive with imaginary super-encryption