The 40th basement door from the left opens to a storage closet until someone says, "swordfish" at the 30th door from the left, then 15 seconds later, it opens to a vestibule with an imposing, riveted-iron door for the next 60 seconds. That locked door requires a genuine invitation to admit you to the speakeasy.
If you didn't know the speakeasy was there, you might not bother trying to dig through the back wall of the closet with a pickaxe. If you watched someone else go in, and copied their actions, you still don't have the invitation. Any noise you make banging around trying to fool the automated bouncer is much more noticeable when all the casual traffic and robot-driven attackers are mostly just stealing boxes of detergent out of the decoy closet.
For example, say that I not only move ssh to port 24, but it's also completely disabled by default. Then I have a small script scanning icmp logs looking for a ping of a particular size on another obscure port, and if it gets one, it enables the ssh server for 30 seconds. If no one opens an ssh connection in that window, it re-disables.
How would anyone besides an insider even figure out how to enable your ssh port let alone try to break in? Sure, if this method became widespread the script kiddies would adapt accordingly and it would no longer be as effective, but staying one step ahead of the kiddies is pretty easy.