You also need to add "PubkeyAuthentication no" to your global stanza, and re-enable it for good hosts.
Or rename the id_* keys to something else. If you're using multiple key pairs will want more descriptive names anyway.
Re:Show HN: My SSH server knows who you are
by chrisfosterelli on Tuesday August 04, 2015 @05:55PM
(#10006528)
Ah, good catch! I don't normally name my keys after the default "id_rsa" so I didn't notice that. You will need that configuration you showed if you have keys that use the default value.
The result is that with this configuration you would still send id_rsa to unknown hosts.
You also need to add "PubkeyAuthentication no" to your global stanza, and re-enable it for good hosts.
More instructions: https://github.com/FiloSottile/whosthere#how-do-i-stop-it