I'm actually shocked that you're the first one to mention it. Only use public key logins and if you're not targeted and if it's practical in your workflow change the port (definitely not for security, only so that sshd won't eat as much CPU time from bruteforce attempts).

Just make sure you don't use a port any higher then 1024 or else a non root process can start up and take its place if the real one crashes.

Oh, good to know, I will ignore this advice and excitedly await for this to happen to me

That's an excellent point. I painted myself into a corner here because I am letting some of my friends use the server as well. Have to gently transition them to public key usage only now.

Give them a reasonable deadline, and just disable password login afterwards.

This isn't something they should have trouble understanding/accepting.

You can restrict password auth to particular users/groups, which also greatly reduces the attack vector.

Or require OTP for password authentication.