Also, you can put port numbers in your SSH config, so you never have to type them.
The only thing moving the SSH daemon does is give us a slight, conditional performance benefit (that using fail2ban on port 22 also gives, along with an actual security benefit). It does not give us any security benefit at all.
On the other hand, your explanation plus the other replies probably help people dispel this notion for others, so maybe it's good to keep this kind of thing coming around.
All the change of port does is reduce the load avg from brute force attacks and failed connection attempts. You can accomplish the same thing with a fail2ban rule on the standard port.
Changing the port doesn't prevent intrusions. That's not a security benefit. That's a performance benefit, and a short-lived one at that, since what do you think botnet hackers will do once they've figured out that we're all running SSHd on a non-standard port and their brute force success rate goes to zero? They're going to adapt to the situation and scan the machine before connecting to it.
It's not security if it doesn't give a guarantee. Changing the port guarantees nothing.
I typically don't bother, and just use fail2ban, but it's silly to say that it doesn't make it more secure. It does, if only marginally. As long as people are aware that it isn't sufficient, and that they still need to use strong passwords and/or public key auth, then there's no problem with saying that this makes things more secure.
You shouldn't mistake obscurity for security, but obscurity that doesn't actively bother the user doesn't hurt security. Again, emphasis on not mistaking obscurity for security, i.e. moving SSH to a port other than 22 does not mean that you can keep using password authentication.
Virtually all DNS servers on the Internet deny zone transfers (AXFR). This is entirely an "obscurity" measure, but it's still useful as long as people don't assume one of their hosts won't be found if it has a guessable hostname or IP address. There are many other examples.