If someone can read the files in .ssh, chances are they can also add an alias to the ssh command that steals your passphrase. As for the "stolen laptop" scenario, whole disk encryption is preferable.