Switching SSH to any other port is a good idea as it will reduce the random attacks against it.

Sure, those drive-by attacks are pretty weak and not much of a threat against a hardened configuration and the real attackers will find the new port anyway, but it reduces noise significantly.

Cleaner logs are easier to parse, so the net result is that you can spot attacks that you care about much more easily.

Install fail2ban, leave ssh on port 22, and have clean logs.

If someone knows your IP, can't they spoof the packets and get you banned from your own server? Or did they solve this issue somehow already?

The ssh authentication works over TCP which isn't realistically spoofable because to open a connection (which you need first, before you can fail to authenticate ssh), you need to be able to receive the response packets to your spoofed sender address

a) This was suggested as a method to combat random trolling from botnets, so this sort of response is highly unlikely unless you're already getting DDOS'd [1], and b) Fail2Ban allows you to whitelist some IP's that won't be banned, if you really need to protect against that sort of attack.

1: at which point you'd most likely need to be a lot more involved anyways, and you should most likely be running something more serious in front of your server anyways...

there is a whitelist capability