If you only support public key or two factor authentication you won't need to worry about brute force attacks. Most SSH brute force attacks are dictionary based using common usernames.