Not from my experience. If you have too many keys and certain ssh agents like gnome keyring don't pick up the key intelligently and will try a all the keys in some order often resulting in the server giving rejecting you due to too many failures.
But you don't live with it -- you either move the extra keys to subdir, so that gnome keyring does not pick it; or use "IdentitiesOnly yes"/"IdentityFile foo" in .ssh/config to restrict certain hosts to certain keys (and yes, those work with ssh agent caching too).
I know many people just don't care about working tool, and tolerate the pain, but hopefully if someone knows enough to setup fail2ban, they should also be able to setup ssh config. Especially since reliable ssh connections is such a high quality of life improvement.
However I think it's a good habit to make records in `~/.ssh/config` for each of your servers anyway just to keep tabs what, where, who, and with what keys.
I move my port somewhere else to reduce the clutter
> manually look at the logs every once in a while
I imagine that this is more by curiosity than anything else (which is a perfectly normal reason to do that - I do it from time to time just because). For logs analysis and alerting I have automated systems.
> using password auth is a bad idea
It is not if the password is correct, security speaking.
If you are logging in via ssh, the chances of being locked out arr low - using password auth is a bad idea, and once you set up ssh keys, the connection will always succeed. And in case of rare event like new system setup, you can always ssh via some other system, -J is great for that.