and then all you'd have to do is compromise SSH on my fully patched system

of course the point is not to be able to withstand an attack by a persistent attacker. it prevents my logs from filling up with the constant scans from scripts