It is almost as bad to have your passphrase-protected key permanently stored in ssh-agent, because anyone with access to your machine can use the key without the passphrase. A better solution is to use ssh-agent with the -t option to establish a lifetime (after which you will need to re-enter the passphrase).

My setup is to keep ssh-agent running with a 2-hour lifespan, and connect to that automatically when I log in. Basically this:

    $ ssh-agent -t 7200 > ~/.ssh-agent
    $ echo "source ~/.ssh-agent" >> ~/.bash_profile
    $ [... log in ...]
    $ ssh-add

I am not annoyed by re-entering my passphrase every two hours.