News for hackers. Stuff that matters.
Re:Post-mortem and remediations for the Matrix.org security ... by Arathorn on Wednesday May 08, 2019 @07:22AM (#19857776)
Author here - hopefully the level of detail here will let others learn from our mistakes (and hopefully benefit from how we've chosen to fix them going forwards). Happy to answer any/all questions or comments.

TL;DR: keep your services patched; lock down SSH; partition your network; and there's almost never a good reason to use SSH agent forwarding.

Re:Post-mortem and remediations for the Matrix.org security ... by voltagex_ on Thursday May 09, 2019 @01:07AM (#19865566)
>The attacker put an SSH key on the box, which was unfortunately exposed to the internet via a high-numbered SSH port for ease of admin by remote users, and placed a trap which waited for any user to SSH into the jenkins user, which would then hijack any available forwarded SSH keys to try to add the attacker’s SSH key

You could also fund/donate to/advocate for a better SSH agent.

I use both Pageant and ssh-agent in my home network for ease of ssh'ing into boxes, especially Unifi gear and some dev VMs. I don't think I will stop using agents, but I probably wouldn't use them at work.

Why couldn't there be an agent that required you to touch a Yubikey before it'd allow keys to be forwarded? Why couldn't you add prompting and timeouts to an agent?

Re:Post-mortem and remediations for the Matrix.org security ... by 32032141 on Thursday May 09, 2019 @01:34AM (#19865668)
Just use ProxyJump. You basically should never be using agent forwarding.
Re:Post-mortem and remediations for the Matrix.org security ... by JoachimSchipper on Thursday May 09, 2019 @01:21AM (#19865618)
ssh-agent has prompting and you can set up a Yubikey with ssh.

The problem here was agent forwarding, which you should almost always replace with opening a new connection via ssh -J (or equivalent.)

Re:Post-mortem and remediations for the Matrix.org security ... by voltagex_ on Thursday May 09, 2019 @01:50AM (#19865731)
But can I prompt every time the agent is used?
Re:Post-mortem and remediations for the Matrix.org security ... by Arathorn on Thursday May 09, 2019 @01:56AM (#19865750)
How would you know whether the agent is being used by a legitimate app or a malicious app racing with a legitimate app to steal access?

At least you only would leak a single access, and you would have a higher chance of noticing, but I can also see that if the hijack was done intermittently you might write it off as a glitch...

Re:Post-mortem and remediations for the Matrix.org security ... by jkire on Thursday May 09, 2019 @08:27AM (#19867617)
Yup, if you're using ssh-agent (as opposed to something like gnome-keyring) setting `AddKeysToAgent confirm` to your ssh config should cause a pop up to happen every time anything requests a key from the agent.
Re:Post-mortem and remediations for the Matrix.org security ... by vector_spaces on Wednesday May 08, 2019 @11:40PM (#19865173)
Thanks so much for this write-up, really appreciate the candidness and detail.

I've learned a lot from it and will be adding some of these practices to the infrastructure that I manage.

Re:Post-mortem and remediations for the Matrix.org security ... by Arathorn on Wednesday May 08, 2019 @09:49PM (#19864668)
I should probably also link to the HN thread from when this happened, which has a lot of interesting discussion about the security issues in question: https://news.ycombinator.com/item?id=19642554

Also, https://news.ycombinator.com/item?id=19643227 is an excellent tl;dr.