> I have never knew a single person put ssh host key into HSM.

You probably also never met a single person where the SSH interface sees millions of sessions as day with valuable assets (code) being transported over said sessions.

> In fact, this is not a supported option in openssh.

This definitely is supported. Though documentation for this is often HSM vendor specific, which if heavily NDA'd. So that's why you probably haven't found much information about it.

What I expect has happened here is that you've remembered that your HSM comes with instructions for how to use PKCS11 to make user authentication rely on the HSM and you've just assumed that's relevant here. While I'm sure the vendors make it seem like this is all very secret, it's just a pretty boring C library and it's probably half-arsed in real world implementations.

AIUI OpenSSH does not provide any way to use PKCS11 to protect host keys, which are the concern here.

You can use PKCS11 to sign OpenSSH certificates, so if GitHub had elected to use certificates here, it could have protected the CA keys for those certificates in an HSM, but it did not.

Correction: It was pointed out elsewhere that you can just tell sshd to use PKCS11 keys via the SSH agent mechanism, and so yes that would allow use of an HSM for host keys

> This definitely is supported. Though documentation for this is often HSM vendor specific [...]

How can openssh documentation be vendor-specific?

Or are you saying that vendors commonly provide an openssh fork/patchset/plugin allowing for HSM-resident host keys?

> How can openssh documentation be vendor-specific?

I isn't, because the cryptography is (in case of HSM) not handled by OpenSSH itself. So OpenSSH's configuration has nothing to do with the HSM.

Usually, the actual cryptographic functions are not performed user-space, but handled by the kernel, which in turn can offload this to dedicated hardware. Basically if you compile OpenSSH for it to use kernel level TLS, then OpenSSH can work with a HSM without it even knowing it.

Disclaimer: this is simplified explanation, there is a lot more to this, and I am by no means an expert on this matter.

So you‘re saying that OpenSSH has an interface for that on the host key side?

I‘m aware of the PKCS#11 integration in the OpenSSH client and have dabbled a bit with it but was not aware of any server side equivalent.

And how does TLS fit in here? SSH is a very different protocol from that, no?

Update: I can't find any OpenSSH documentation of either (server-side) PCKS#11/HSM support or kernel-mode cryptography (which also in the case of Linux only addresses symmetric encryption to my knowledge, at least the mainline kernel version I know of).

Maybe you're thinking of some other SSH implementation? The protocol definitely allows for server-side HSM usage, and Github at their scale is not bound to OpenSSH in any way.

It's just the agent protocol, used by sshd instead of ssh, to make signing requests with a host key (instead of a user's identity key).

That's cool, I wasn't aware that the server supports the agent protocol as well. Thank you for the pointer!

It makes a lot of sense, since it avoids having to link the HSM/PCKS#11 stuff against sshd.

> I can't find any OpenSSH documentation of either (server-side) PCKS#11/HSM support

OpenSSHd talks to an ssh-agent that then talks to the HSM:

> Identifies the UNIX-domain socket used to communicate with an agent that has access to the private host keys. If the string "SSH_AUTH_SOCK" is specified, the location of the socket will be read from the SSH_AUTH_SOCK environment variable.

* https://man.openbsd.org/sshd_config#HostKeyAgent

Interesting, I didn't know that OpenSSHd supported the agent protocol. Thank you!

Why is everyone just authoritatively dismissing this, when this has been supported for >7 years and is easily found with a google search?

There is the HostKeyAgent configuration directive, which communicates over a unix domain socket to make signing requests.

https://framkant.org/2017/10/strong-authentication-openssh-h...

https://github.com/openssh/openssh-portable/blob/12492c0abf1...

> This definitely is supported.

Agreed. I have seen some crazy stuff in the payment card industry. I can't recall what I can and can't talk about so I'll just say "Atalla".

Yes, but that would either be a fork of OpenSSH, private or open source (both are possible since it's BSD-licensed), or a different SSH server (which Github is of course free to use, since the protocol is standardized and their scale absolutely justifies any efforts in protecting their SSH host key). But GPs comment was about OpenSSH.

It would work with OpenSSH's HostKeyAgent option.

There is the HostKeyAgent configuration directive, which communicates over a unix domain socket to make signing requests.

https://framkant.org/2017/10/strong-authentication-openssh-h...

For what it's worth, Github uses libssh (https://www.libssh.org/) for their ssh servers.

It looks like they currently use the `ssh_bind_options_set` function with `SSH_BIND_OPTIONS_HOSTKEY` to set the host keys which means they exist on disk at some point. HSM aside, I believe it would be possible to use the `ssh_bind_set_key` and deserialize them from a secret vault so they only exist in the memory of the ssh daemon.

Obviously they also just straight up have enough resources to fork the code and modify it to use an HSM.

Source: looking at their ssh server portion of `babeld` in ghidra right now as part of hunting for bug bounties.