A Yubikey-like device is actually more secure than what the parent comment describes, because it prevents the private key from being viewed or copied by the host computer. You can think of it as a dedicated piece of hardware that runs something like ssh-agent, except over USB instead of a socket.
Aren't you describing Yubikeys? Which aren't widely supported, but IIRC are still supported by some websites and apps.