This is a nifty trick, but I recently had an issue where it would offer my other keys even with IdentitiesOnly yes specified in ~/.ssh/config. I found a solution [1], but it's not the most elegant.

[1] https://superuser.com/a/859719