While other posters are right to point out that server certificate should make sure nobody can truly hijack your sshd there's no point in taking the risk unless for some reason all privileged ports are in use.
Not to mention that ssh's key model is a bit broken since there's not built-in way to distribute the keys/check the keys against an authority like with SSL. Most people just accept the key they're given when they first connect (I know I do).
I think the recent NSA news indicate that the SSL authority model is broken. Any 3rd party authority can be subverted by legal (e.g. NSL) means.
SSH's model leaves that only semi-solved. you CAN distribute your own "known_hosts" file, thereby avoiding the need for either an SSL-style authority, or remembering every host's key. Alternatively, you can use something like SSHFP+DNSSEC/DNSCurve or Monkeysphere if you like the underlying trust models.
ssh's key model is NOT broken. SSL's models IS practically broken. ssh just leaves a little less specified.
http://web.monkeysphere.info/getting-started-ssh/
So I figured I'd share it here: https://news.ycombinator.com/item?id=6617132
If you want actual more protection run a bogus SSH (with no login allowed) on 22 and a thousand unused ports... then they have to try them all or guess which port is actually able to log in.
You are right that it doesn't protect from a targeted attack. In my experience (10 years as a sysadmin and dev), I've never had a targeted attack -- all the attempts against my machines are drive by.
Having port 22 open can end my IPs on a list of "let's try to break later".
It's not close to a complete solution, but I find it beneficial.
Everyone who's expecting to see HTTPS will see it on :443 as well.
The solution I'm particularly fond of is mentioned in Daniel's blog comments, which is to use iptables to block externalip:22 and redirect a non-standard port to localhost:22.
I run my SSH daemons on port 1022. It reduces brute-force attacks significantly without reducing security in any way that I'm aware of. I also have a ~/.ssh/config file in my laptop that tells SSH to use port 1022 whenever I'm connecting to one of my own remote machines so I don't even need to type anything extra when I use ssh, rsync, etc.