~/.ssh/id_rsa can be protected by a password, you'll want to access the running ssh-agent process memory to get the key in cleartext (unless the person use some security token in which case you can't access the key, just try to login on a remote computer directly while the HSM is available).

But while ssh-agent has the key unlocked on a Linux system, any process running as that user can use the key without knowing the passphrase the ssh-agent. That's the more direct comparison.