> still doesn't matter (...) you could literally have a complete RAM dump of the whole system and not be able to access it.

I'd say that matters. Think about all the secrets (tls keys, whatever) a server has in memory.

If you can't connect to the sshd daemon, you can't attack it.