One thing that always has scared me a bit with using CAs for SSH is how you protect the signing certificates? After all, if an attacker gets that cert then they get full access to everything, and can masquerade as anyone. You end up with a choice between a) have lots of SSH keys out in the wild, each with varying degrees of access, or b) have a single cert that is on your infrastructure but has access to everything. (Not to mention how you deal with the operating the site, what happens if it crashes? How do you log in without the site to sign your ssh key? Using standard trusted SSH keys to access feels like its somewhat undermining the point of using CAs).

Has anyone solved this, or got a write up of some best practices for running this? All I've managed to find are articles about how to run such apps, rather than how it fits into the broader security architecture.

Ideally ideally, what I would actually like is the ability to configure OpenSSH to require multiple things to log in, i.e. both that they SSH key is trusted and that it has recently been signed by the signing service. That way gaining access to the signing certificate doesn't help without also gaining a trusted SSH key (it's still bad, but not quite Game Over levels of bad). I had a quick look to see if I could hack together a patch to do this, but alas I had forgotten how weak my C foo is :(

> how you protect the signing certificates

You get an HSM like this: https://www.veritech.net/product-detail/keyper-hsm/ that stays air-gapped.

Then you build procedures around it, like https://www.iana.org/dnssec

Not cheap or easy.

If you have no compliance requirements, you can also just use any pkcs#11 token (with support for non-extractable keys) to secure the key, and setup an air-gapped process on a laptop with a bootcd, etc, to minimize the risk of compromising your process.

> ... what I would actually like is the ability to configure OpenSSH to require multiple things to log in, ...

With OpenSSH, you can require multiple authentication methods to succeed before access is granted.

For example, "publickey,password" to require password authentication after key-based authentication has succeeded. You could even do "publickey,publickey,publickey" to require three different keys to be used!

This has been supported for several years, by the way. See "AuthenticationMethods" in the "sshd_config*" man page.

Have you found any such write-ups on how this all fits together - I am also looking in vain.

> ideally from a strong root of trust, like an IdP that does 2FA.

I understand the concepts, but how does this work in practice? Do you have an example of generating a short-expiry certificate from an IdP, such as Google?

You can do it directly with OpenSSH, no need for third-party software. There are many good blog articles / tutorials on the subject, e.g. search for "ssh ca certificate". Most people don't know that you can do this but it's actually quite easy.

I did a bit of reading on the topic. But it is still unclear to me what the workflow is. How would a typical day look like for an admin and one of the users?

Vault can do something like this.

We do that with hashicorp vault and adfs+duo it works pretty nicely.

The question that can be asked then is: how often should should I rotate the CA key? ;)

You never rotate it directly. Instead, generate it from your SSH keys.