It is as simple as reducing an attack surface. If attackers can't talk to sshd, then can't try to hack it. In a world where zero days are real, why chance it?

Why is that so hard to grasp. Still boggles my mind.