Working as a sysadmin, and with disk encryption the constant log writes generated from ssh has been enough to cause some heavy load on severs. Long lasting connections from bots also forced is to increase the maximum number of sockets. Fail2ban has mitigated those issues to the point where neither has been an issue.
I believe this depends on your exposure level. If you have one, or maybe no, domain names pointed at your server, you are probably fine. If you have tens, or hundreds, of domains pointed at your server (i.e. you are a web site hoster), you will probably need something like fail2ban.
On another server with the same setup, there's various services and websites running, so load average is never zero there (due to scrapers) but ssh has yet to catch my eye in htop. Whenever I've looked, even the amount of cpu that a Windows VM uses constantly while idle is many times higher than what ssh uses
What cpu and especially what disk load are you talking about? Also not running out of sockets, not sure if that's RAM-dependent (the box has 2GB)