A better mandate would be to use ssh-agent. Using separate key pairs without an agent doesn't prevent key theft due to memory disclosure vulnerabilities like this one; it merely compartmentalizes the damage. But if you use an agent, the client never loads the private key, so it's safe from memory disclosure vulnerabilities. Stealing a private key would require filesystem access or remote code execution, at which point all of your private keys are at risk, so you might as well just use a single private key per device.

For the best security, store your private key on a smartcard, so even an attacker with filesystem access or remote code execution can't steal it.

This is what I would assume, but has it been confirmed that this vulnerability doesn't affect identities provided by an SSH agent? If any of this "roaming" support was provided by the agent, it could be the case that the leakage can be triggered via SSH agent protocol requests. I doubt this is the case but I don't see anything confirming or denying it.

It would be nice to hear some official word on whether agent-based authentication is vulnerable, because if it isn't, the seriousness to me is greatly reduced, as I never load identities in the client directly.

https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-077...

Quoting from above: Finally, for these three reasons, passphrase-encrypted SSH keys are leaked in their encrypted form, but an attacker may attempt to crack the passphrase offline. On the other hand, SSH keys that are available only through an authentication agent are never leaked, in any form.

So if you use an agent, and follow the good advice to encrypt your private keys you should be safe(er).

See for older client versions:

http://martin.kleppmann.com/2013/05/24/improving-security-of...

or better for newer clients:

http://www.tedunangst.com/flak/post/new-openssh-key-format-a...

https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-077...

This means they are basically able to dump the memory of just the running `ssh` process -- eerily similar to Heartbleed.

This means private keys stored by the `ssh-agent` process, outside of the `ssh` process connecting to an Evil Server(TM) are not affected.

This is because the protocol used between SSH Agent and SSH client does not transfer the entire private key, rather the SSH client asks the Agent to do a signing operation on it's behalf.

Make sure you patch BEFORE doing something like ssh'ing into whoami.filippo.io "just because someone on the internet told me to"

This site is (from what I can tell) not malicicous, but copy pasting scripts and blindly following advice from the internet is a prime example of how an exploit would occur.

  Host *
  IdentityFile ~/.ssh/%h

    # IdentityFile magic, should be placed at very end of file
    Host *
    IdentityFile ~/.ssh/keys/id_ecdsa_%r@%h
    IdentityFile ~/.ssh/keys/id_rsa_%r@%h
    IdentityFile ~/.ssh/keys/id_ecdsa_ANY@%h
    IdentityFile ~/.ssh/keys/id_rsa_ANY@%h
    IdentityFile ~/.ssh/keys/id_ecdsa_%r@ANY
    IdentityFile ~/.ssh/keys/id_rsa_%r@ANY
    IdentityFile ~/.ssh/keys/id_ecdsa_ANY@ANY
    IdentityFile ~/.ssh/keys/id_rsa_ANY@ANY

Edit to add: While I was testing this understanding (which is correct) mioelnir's comment added the setting you need to get the behavior which I thought was automatic.

    IdentitiesOnly yes

  Host hostB
    ...
    ProxyCommand ssh hostA -W %h:%p

1) from your client to hostA

2) from your client to hostB

So to answer your question - no, vulnerable client on hostA is not a problem (or at least not in this particular use-case).

Ok, that's what I was wondering. Thanks.

https://heipei.github.io/2015/02/26/SSH-Agent-Forwarding-con...

There's places where there are two alternatives: agent forwarding or copy the key. Using ssh agent forwarding allows me to use a smart card for the key. It still has it's problems but it's much better than having the key on the remote machine - for example the use of a smartcard mitigates this vulnerability since the key never enters the process memory.

Note that for ProxyCommand to work, you don't need a full shell on a jumphost, just "AllowTcpForwarding yes" is enough. On the other hand, with AgentForwarding method you do need a full shell on a jumphost.

Of course local, not forwarded ssh-agent on the terminal would be super-handy to avoid typing pass-phrase time and again; but that's different and independent from ForwardAgent.

What do you do now if someone leaves? Remove that person's key from all 50 hosts one at a time?

Or, at the very least, you use tmux with sync panes or csshx - and log into all 50 hosts at once, and then you can issue one rm / scp command if you are still doing things manually.

Are my guys going to spend 5 minutes every morning making and pushing keys?

What if my hosts auto provision themselves and there are 5 new hosts every morning?

Am I going to make keys as part of infrastructure deployments and push them back to the workstations and update other peoples ssh configs?

I'm just saying if you've worked at scale, you'll realize that a key per box won't scale. I mean, anything can scale if you put enough effort into it. But the chances of disaster in lack of access or security breach, from over complication is way to high here.

Automating bad processes just makes it easier for them to fuck you.

Even with config management this won't scale past about 2 or 3 people and 10-20 boxes.

Central auth is an option I guess but I think the better way to go would be a 2 factor with the key and hotp.

OpenSSH can also be used in a PKI fashion, where you use certificates instead of known_hosts and authorized_keys records. It's quite all right, but it comes with the same problems a full PKI does as you need to keep track of when the certificates expire. You also need a way to distribute CRLs so you still need configuration management.

As patio11 notes, this would not at all be excessive as a response to the threat model, but even with an aggressive key rotation mandate, this still doesn't make your production infrastructure as secure as your corporate Confluence wiki with SSO. Your employee's private key is still a static credential, and any rotation policy (14 days? 30 days? 90 days?) will leave a significant window for an attacker to use a stolen credential. Additionally, using a single key per employee for all infrastructure access magnifies the attack surface of a stolen credential to everything you operate.

Full disclosure, I'm a co-founder at ScaleFT, a startup focused on solving these sorts of problems. We're releasing a patch to fully mitigate this for our users this morning.

The reasoning is that it is far easier for me (or the entity, but that's a bit different) to delete one keyfile to sever access than it is for me to rekey everything else. I don't want access to things I'm not actively engaged with - compromises happen, even to engineers' laptops, and that conversation with former employers is too much like calling up your exes to tell them about a VD test result for my comfort.

As far as it being digital identification, lots of companies have IDs separate from your DL/passport. This is usually because the company ID provides access to something your other ID doesn't. Same principle.

[1] There are lots reasons to have lots of different keys, and only having one per entity is pretty rare for me.