Often the mentality you will find is "any hole is a hole that will be exploited", so they make these incredibly difficult systems with perfect security that nobody ever uses because they're so hard to use.

SSH is flawed in that you have to trust the host the first time you connect to it. For most people this is a reasonable tradeoff, but for security nerds it can be an impossible barrier to adoption. This is why some SSH servers spit out big long hexcodes or weird picture things, you are supposed to securely message the owner of the server to verify it before you accept it. Details are of course left up to the user because any default system you set up will of course be exploited. It's head of line security exploits all the way down.