Under Windows, you can forward your smartcard over remote desktop. It's one of the few things Windows has I miss on Linux.
To your point, I wonder whether that consideration holds when the private key is held on an external device, like is the case with a YubiKey. I use that setup, and I can't add the key to the ssh agent.
$ ssh-add .ssh/id_yubikey_gpg.pub
Error loading key ".ssh/id_yubikey_gpg.pub": error in libcrypto
They're specifically refering to password authentication to make the ssh connection.
I prefer to have sudo ask for a password when I'm physically in front of the machine, but not if it's a remote session (e.g. SSH from my laptop to my desktop).
Maybe the SSH agent on the client can re-authenticate to the server when requested?
There is a PAM module that does this: https://github.com/jbeverly/pam_ssh_agent_auth
Note that this is a bad idea from the security standpoint, as it requires SSH agent forwarding. Which means that, if the remote server is compromised, the attacker can use your SSH agent to log into other servers as you.
And your terminal may come with a password manager too, which would be unlocked with whatever means.
Again, on a Mac with iTerm you can do this with a fingerprint.