I'm actually shocked that you're the first one to mention it. Only use public key logins and if you're not targeted and if it's practical in your workflow change the port (definitely not for security, only so that sshd won't eat as much CPU time from bruteforce attempts).