But then it would have to either keep the passphrase in memory (just as bad) or ask for the passphrase each time you reconnect later (defeats the purpose of ssh-agent).

Edit: Nevermind, you're talking about ssh, not ssh-agent..