but the original objection was regarding using passwordless keys. again, the main concern here isn't about automated attacks, it's about targeted attacks. someone got your key, it doesn't matter that you're running on a non-standard port. they'll figure that out, if they're targeting your machine specifically.
Think of it as security through irregularity instead of security through obscurity. You are not making your box secure by changing the ssh port, but you are decreasing the number of automated attempts that will hit your ssh port, which could, in theory, reduce your security risk.