The only thing you really need to do with SSH is to use keys with it, not passwords. That should be secure enough for almost all cases.

Another layer on top is useful to remove the noise from the logs. And if you have anything aside from SSH on the server that doesn't need to be public, restricting it via a VPN or something like that is useful anyway. Most other software that listens on your server has likely much more attack surface than SSH.