All the change of port does is reduce the load avg from brute force attacks and failed connection attempts. You can accomplish the same thing with a fail2ban rule on the standard port.
Changing the port doesn't prevent intrusions. That's not a security benefit. That's a performance benefit, and a short-lived one at that, since what do you think botnet hackers will do once they've figured out that we're all running SSHd on a non-standard port and their brute force success rate goes to zero? They're going to adapt to the situation and scan the machine before connecting to it.
It's not security if it doesn't give a guarantee. Changing the port guarantees nothing.
I typically don't bother, and just use fail2ban, but it's silly to say that it doesn't make it more secure. It does, if only marginally. As long as people are aware that it isn't sufficient, and that they still need to use strong passwords and/or public key auth, then there's no problem with saying that this makes things more secure.
On the other hand, your explanation plus the other replies probably help people dispel this notion for others, so maybe it's good to keep this kind of thing coming around.