I‘m aware of the PKCS#11 integration in the OpenSSH client and have dabbled a bit with it but was not aware of any server side equivalent.
And how does TLS fit in here? SSH is a very different protocol from that, no?
Update: I can't find any OpenSSH documentation of either (server-side) PCKS#11/HSM support or kernel-mode cryptography (which also in the case of Linux only addresses symmetric encryption to my knowledge, at least the mainline kernel version I know of).
Maybe you're thinking of some other SSH implementation? The protocol definitely allows for server-side HSM usage, and Github at their scale is not bound to OpenSSH in any way.
It makes a lot of sense, since it avoids having to link the HSM/PCKS#11 stuff against sshd.
OpenSSHd talks to an ssh-agent that then talks to the HSM:
> Identifies the UNIX-domain socket used to communicate with an agent that has access to the private host keys. If the string "SSH_AUTH_SOCK" is specified, the location of the socket will be read from the SSH_AUTH_SOCK environment variable.
I isn't, because the cryptography is (in case of HSM) not handled by OpenSSH itself. So OpenSSH's configuration has nothing to do with the HSM.
Usually, the actual cryptographic functions are not performed user-space, but handled by the kernel, which in turn can offload this to dedicated hardware. Basically if you compile OpenSSH for it to use kernel level TLS, then OpenSSH can work with a HSM without it even knowing it.
Disclaimer: this is simplified explanation, there is a lot more to this, and I am by no means an expert on this matter.