I think the scary reality is most people conflate "keys" and "certificates". I have worked with security engineers that I need to remind that we do not use SSH certs, but rather key auth, and they have to think it through to make it click.

I'm consistently amazed how many developers and security professionals don't have a clear understanding how PPK even works conceptually.

Things like deploying dev keys to various production environments, instead of generating/registering them within said environment.

One of the worst recent security examples... You can't get this data over HTTPS from $OtherAgency, it's "not secure" ... then their suggestion is a "secure" read-only account to the other agency's SQL server (which uses the same TLS 1.3 as HTTPS). This is from person in charge of digital security for a government org.

Or when the security team at some other company emails you their private key.

LOL, yeah.. had that happen quite a few times... Also, re-using the ssh server key for the client connecting to the sftp server.

> Things like deploying dev keys to various production environments, instead of generating/registering them within said environment.

I can see this happening when a developer is authorized to generate, but not to register. So, they just reuse an already-registered one.

In the example, it wasn't even that complex... I have used patterns to register allowed signer keys based on environment variables that an application runs under, initializing at startup... so "register" just meant assigning the correct values for 2-4 environment variables per public signer allowed... and removing the dev signer. (JWT based auth)

One key technological cause is that PKCS#12 standardizes a format (you've most likely seen it as .PFX files) in which a certificate and its associated private key are bundled. This is in an effort to simplify the software...

So you get a situation where the lay person is given a "certificate" but it's not really just the certificate it's a PFX file and so e.g. no they mustn't show it you, it has their private key inside it and so you will learn that key and if you're honest you've just ruined their day because they need to start over...

I would say in my career I've had at least two occasions where I did that and I felt awful for the person, because I had set out to help them but now things are worse, and I've had a good number of later occasions where I spent a lot more of their time and mine because I knew I need to be very sure whether their "certificate" is actually a certificate (which they can show me, e.g. Teams message me the file) or a PFX file (thus it has their private key) and I must caution them to show nobody the file yet also try to assist them.

Another useful feature of SSH certificates is that you can sign a user’s public key to grant them access to a remote machine for a limited time and as a specific remote user.

The capacity to grant access as a specific remote user is present without certs as well right? The typical authorized_keys file lives under a user directory and grants access only to that user.

The main advantage of certificates is that you are able to do that from the CA without touching the target machine.

Exactly. This is really useful in larger organizations where you may want more complex rules on access. For example, you can easily build "break glass" or 2nd party approved access on demand. You can put whatever logic you need in a CA front-end.

You can also make all the certs short-lived (and only store them in ram).

The way I've been doing that is with Shamir Secret Sharing and encrypting keys until glass-breaking is necessary.

generating tons of keys? or just broad keys?

What I've done is generate a cert for the host(s) the user needs, for the time-span they need (subject to authorization logic).

Certs may still be the right approach, but OpenSSH also supports an AuthorizedKeysCommand which could be a secure HTTPS request to a central server to pull down a dynamically generated authorized_keys file content for the particular user and host.

If your endpoints can securely and reliably reach a central server, this gives you maximum control (your authorized_keys HTTPS server can have any custom business logic you want) without having to deal with certs/CAs.

And when your or someone else's infra down to such a degree that you need SSH access, you do not want to depend on being able to touch that machine first. The same is true with custom AuthorizedKeysCommands that phone home.

I've known SSH certs for a while but never went through the effort of migrating away from keys. I'm very frustrated about manually managing my SSH keys across my different servers and devices though.

I assume you gathered a lot of thoughts over these 15 years.

Should I invest in making the switch?

A big problem I have with ssh carts is that they are not universally supported. For me, there is always some device or daemon (for example tinyssh in the initramfs of my gaming pc so that I can unlock it remotely) that only works with “plain old ssh keys”. And if I have to distribute and sync my keys onto a few hosts anyway, it takes away the benefits.

Might actually be a positive instead of a negative. Gaming use-cases should have not any effect on security policies, these should be as separate as possible, different auth mechanisms for your gaming stuff and your professional stuff ensures nothing gets mixed.

remote unlock is also useful when you're not gaming so that feels like the wrong aspect to focus on

Hah? It being my gaming machine has nothing to do with the problem. It’s also my FPGA development machine, though it gets used less for that. It only happens to be the only Linux workstation in my home (the others are Macs or OpenBSD).

If you care about security, I recommend investing into a separate computer for developing hardware and software and another for downloading games on.

You can setup your security any way you like, but nothing beats an air gap in terms of security and simplicity.

Upgrade to a better one in initramfs?

Adding to this: while certs are indeed well-supported by OpenSSH, it's not always the SSH daemon used on alternate or embedded platforms.

For example, OpenWRT used Dropbear [1] instead, which does not support certs. Also, Java programs that implement SSH stuff, like Jenkins, may be doing so using Apache Mina [2] which, though the underlying library supports certs, it is buggy [3] and requires the application to add the UX to also support it.

[1] https://matt.ucc.asn.au/dropbear/dropbear.html

[2] https://mina.apache.org/sshd-project/

[3] I've been dealing for years with NullPointerExceptions causing the connection to crash when presented with certain ed25519 certificates.

You can just replace dropbear with openssh on OpenWRT. That was one of the first things I did, since DropBear also doesn't support hardware backed (sk) keys. Just move it to 2222 and disable the service.

I reenabled DB on that alt port when I did the recent major update, just in case, but it wasn't necessary. After the upgrade, OpenSSH was alive and ready.

If your use case is such that you are frustrated about managing keys, host or user keys, then yes it does sound like SSH certs would help you. E.g. when you have many users, servers, or high enough cartesian product of the two.

In environment where they don't cause frustration they're not worth it.

Not really more to it than that, from my point of view.

Yes. Caveat: It might not really be worth it if all your infrastructure is managed by these newfangled infrastructure-as-code-things that are quick to roll out (OpenShift/OKD, Talos, etc.) and you have only one repo to change SSH keys (single cluster or single repo for all clusters).

There are some serious security benefits for larger organizations but it does not sound as if you are part of one.

You will have to manage your SSH CA certificates instead of your keys.

The workflows SSH CA's are extremely janky and insecure.

With some creative use of `AuthorizedKeysCommand` you can make SSH key rotation painless and secure.

With SSH certificates you have to go back to the "keys to the kingdom" antipattern and just hope for the best.

Exactly. We'd had discussions about building https://Userify.com (plug!) around SSH certificates, but elected to go with keys instead, because Userify delivers most of the good things around certificates without the jank and insecurity.

It's not that certificates themselves are insecure themselves, it's that the workflows (as the parent points out) are awful. We might still add some automation around that (and I think I saw some competitor tooling out there if you're committed to that path) but I personally feel like it's an answer to the wrong question.

> With SSH certificates you have to go back to the "keys to the kingdom" antipattern and just hope for the best.

Whut? This is literally the opposite.

With CA certs you can create short-lived certificates, so you can easily grant access to a system for a short time.

And what about the CA?

It's no different compared to regular SSH private keys. You need to protect it from compromise.

However, it provides you an additional layer of protection, because it does not need to be on the critical path for every SSH connection. My CA is a Nitrokey HSM, for example. I issue myself temporary certs that are valid only for 6 hours for ephemeral private keys.

Yes it is different. SSH CA keys are harder to secure and attackers have a much bigger incentive to steal them.

You can also configure multiple CA for client auth, and on the client side multiple ca to verify host keys.

I am keeping an eye on the new (and alpha) Authentik agent which will allow idp based ssh logins. There's also SSSD already supported but it requires glibc (due to needing NSS) meaning it's not available on Alpine.

If you mean using OIDC, in that space there's at least https://github.com/EOSC-synergy/ssh-oidc, https://dianagudu.github.io/mccli/ and OpenPubkey-ssh discussed in https://news.ycombinator.com/item?id=43470906 (which might mention more).

How does SSSD support help with SSH authN? I know you can now get Kerberos tickets from FreeIPA using OIDC(?), but I forget if SSSD is involved.

Can't really speak to the point of the guy you're replying to, but the FreeIPA implementation via SSSD does more than just Kerberos tickets. Actually, I think the Kerberos based stuff as it relates to SSH is GSSAPI as part of sshd itself and has little to do with sssd, though I could be wrong.

That said...

If I'm remembering things correctly (and it's been a looong while since I've played with this), FreeIPA's client configures sshd with an AuthorizedKeysCommand that executes a program that queries sssd for the list of authorized keys for a given user. Sssd then uses a plugin to query the LDAP server @ FreeIPA to get the list of keys.

There's also SSHFP (I think) records in DNS if you're using FreeIPA's DNS servers. These provide the host keys for servers for your ssh client to check against. Not sure if that's integrated into ssh itself or something else -- I can't remember how it's implemented offhand -- but it's fairly nifty since you never see the TOFU prompt (or it would be if DNS was actually secure, anyway).

Yes, FreeIPA is Kerberos+LDAP+X.509 CA, and GSSAPI is in OpenSSH (normally with the key exchange patch). SSSD is a local mechanism, not network authentication. I mentioned authorized keys distribution mechanisms elsewhere, but I was thinking authentication (c.f. OIDC), not authorization.

I haven't used SSSD due it not being available for Alpine but doesn't it provide authentication via pam_sss ?

Yes, but its authN components only act locally, and PAM is optional for sshd. It can/does call out to network services like Kerberos/LDAP given a password, of course, but I was thinking of network authN connected directly with OIDC somehow, for which I don't know a mechanism in vanilla OpenSSH. (I don't know what Authentik does for this -- I could imagine it's behind the scenes somehow.) I should probably look it up sometime.

My understanding is since it's an agent running on the target, possibilities will be quite extensive. But it is relatively new and there is no stable release of it yet.

https://docs.goauthentik.io/endpoint-devices/authentik-agent...

It depends on what you want to do. CA certs are easy to manage, you just put the CA key instead of the SSH public key in authorized_keys.

They also provide a way to get hardware-backed security without messing with SSH agent forwarding and crappy USB security devices. You can use an HSM to issue a temporary certificate for your (possibly temporary) public key and use it as normal. The certificate can be valid for just 1 hour, enough to not worry about it leaking.

oh man, I referred back to your blog post when I wrote the ssh certificate authority for $job ... ~10 years ago.

Thank for writing it!

maybe it keeps getting rediscovered, becaues a good tooling around it is still missing?