You put a wall around your property. It kept the dilettantes out, which means you have free time for something else.

If there’s a buffer overflow attack against SSH then you have security through obscurity. In the meantime you have security and obscurity. Assuming you’re using known rsa keys to login remotely...