Oh I mean I know how to do it, I just mean I'd like to see companies talking about how they do it, and I'd really really like to see someone make it drop-dead simple.

It's valuable to hear how companies are protecting their infrastructure if they're going to break down how they accomplished it, how they continue to maintain it, etc.

We use symops[0] to connect to our servers. Not sure about the implementation details, but the gist is that we connect through our aws credentials (via SSM), which we retrieve using Okta, which is then protected using 2fa. No ssh key to be found, and you can disable all ssh ingress rules. Since this is via SSM, you can also use the full power of AWS IAM to allow/deny access.

Also, Sym can be set up to require approvals too which is great for security auditing since it's a third party.

[0] https://symops.com/

Ah nice. Yeah, SSM is awesome, we use it as well.

OpenSSH 8.4 added support for FIDO2. It's literally the same commands (with some different parameters) to take into use as a normal keypair. And FIDO2 tokens are becoming quite affordable. To me 2FA with SSH became a solved problem with that.

Could you point to a relevant article explaining that setup please?

This one had quite a lot of background, but also the needed commands: https://www.stavros.io/posts/u2f-fido2-with-ssh/

With a properly set-up token, basically you only need 2 commands:

   ssh-keygen -t ed25519-sk -O resident -f ~/.ssh/id_mykey_sk
   ssh-add -K

I used a Yubikey and needed also to install 'ykman' to set a PIN for my token, otherwise ssh-add kept failing. Dunno if I omitted something for a proper setup for my token initially, but I don't think that was a problem with OpenSSH in particular.

Apart from the small headache with the PIN, the whole thing was almost magical in its simplicity.

The token must be pin protected to be eligible to resident credentials. I think it is one of the significant differences between FIDO1 and FIDO2. I also think that you can use ed25519-sk with a non pin-protected, but then you won't be able to authenticate if you can't access to the generated key file (if it was deleted or on another machine)

Thanks for the info. Makes complete sense to require a PIN with resident keys, I'm glad. Now I'll need to write the article myself, with this information included :)

If I don't know how to do it, where do I start to understand ?

Edit: I suppose I am asking how does it "all" fit together. The CA that grants servers or services a short lived key - the other servers that then can trust that. It makes kind of sense but I think I am missing some parts as when I try and read how others do it, some parts seem to be missing.

Too many blog posts seem to be something something kubernetes will arrange it. Or Oauth or ...

For example, this article seems to blow against the idea of a central key management service like Vault and have the device decide to rotate keys. But I am not 100% sure because it's one sentence. And how do they provide authentication for a service account (say a web worker that processes some incoming requests). That's not device+person. The same idea can happily apply but do they?

I think I am just moaning.