Using a short ban time keeps the load of running fail2ban to a minimum. I at one point was banning the kiddies/bots for a week and 40% of the server load was chewed up just running fail2ban.
The solution I came up with to this problem is to use a custom ban action that adds an IP to the blocked ipset list with a given TTL, but set fail2ban to think that the IP has to be only banned for 5 seconds or so.
The result is that ipset purges the IP from the list after the TTL expires, but fail2bans working set is a 5 second rolling windows of blocked IPs.
This works great and it's being used in production for a couple of years.
> so that my logs don’t fill with a zillion pointless attempts
You don't really need fail2ban at all. Just don't log failed password attempts. Key access only, no one is going to get in.
Of course there are ways to deal with it, but that's just extra work.
When I was young I too used to try implement detection and banning but realised it's a waste of time. Key only access to ssh and make sure you at least subscribe to security updates for all packages is enough.
For our own app we went the ELK route due to the number of log messages coming from our own application. But yes as you said it's a pain to manage. Also creating kibana graphs/views/desktop for each type of log is also painful.
A good service I believe would be someone to offer a turnkey/shrink wrapped managed ELK for self hosting. Not only managing the system side of it, but also offering services to help create custom kibana graphs.