This is exactly what I do -- I posted some links to both the SSH Agent (well, the ChromeOS version -- I've got C versions as well) and the PKCS#11 module which talks to the SSH Agent. So on a remote system I can do things like sign files using PKCS#11 and passwordless sudo using pam_pkcs11.