Just disable password authentication, problem solved. Changing the SSH port does not add security against people who are determined to break into your system, but as you said, it helps to keep the noise down so you're more likely to notice suspicious activity early.