You realize you've effectively done the same thing as saving the password on the machine to auto-log-in with a password, right? The exception being it's harder to brute force the server (but you did mention server filtering, so forget that). They have ssh-agent for a reason.

p.s. using a password with keys is two factor