My thoughts as well. You can set some core commands to be different in your per-project git config which I find more flexible for SSH identities than using a the .ssh/host file.

https://git-scm.com/docs/git-config#Documentation/git-config...

    git config core.sshcommand "ssh -i /path/to/project/keyfile"