At some point, to gain any form of remote access you have to put your trust in one piece of software. Being it VPN, ssh, or something.

For many people ssh is considered secure enough. And any secure issues get immediate attention so you can be updated pretty quickly.

Now any software could have unpublished zero days, which is why I also do some other host based security.