News for hackers. Stuff that matters.
Re:Night of a cascading failure by aftbit on Monday January 21, 2019 @07:12PM (#18964863)
The most interesting thing here is that this took down the bastion boxes that allowed PROD SSH access. I would argue that was the biggest red flag by far. As the author points out, if you can't access the machines somehow, you are pretty screwed. Great job that you managed to get back in anyway, but really the production access mechanism should be as dumb, simple, and standalone as you can make it.

We use multiple bastion hosts with user accounts provisioned by Ansible. There is one in each data center, plus a "shell" box at AWS that lives outside the firewall and also has SSH access to everything we need. All the bastions can access every server.

Plus, some of the mission-critical boxes actually have SSH exposed to the public internet. I would much rather take the security risk with (key-only) OpenSSH than the risk that I get locked out of PROD when I need to fix things after a 2am page.

It is entirely possible to make a box so secure that you yourself cannot access it, while not actually doing much to defend against more "reasonable" security risks, like typo-squatting and spear-phishing. It's all about the threat model.

Re:Night of a cascading failure by Waterluvian on Monday January 21, 2019 @08:42PM (#18965294)
Your perspective on the less secure but accessible public internet SSH box is interesting to me. It's as if you could wire up a weaker entryway with a ton of alarms of all kinds and probably feel pretty secure that way.
Re:Night of a cascading failure by sterlind on Tuesday January 22, 2019 @02:07AM (#18966557)
Heck, you could have an actual siren connected to the USB port! It'd be annoying enough that people would investigate if anyone tried anything.
Re:Night of a cascading failure by RyanShook on Monday January 21, 2019 @11:45PM (#18966056)
Agreed, I wish the author would have explained why the authentication system was tied in so directly to this bug. Seems like the biggest learning would have been to separate the two.
Re:Night of a cascading failure by hnzix on Tuesday January 22, 2019 @12:45AM (#18966243)
TFA did explain it. The boxen were authenticating against a LDAP server in prod, and prod was dead. This was obviously a huge design flaw and the author identified it as such.
Re:Night of a cascading failure by throes_death on Tuesday January 22, 2019 @12:25AM (#18966182)
> learning

I'm confused, did you mean lesson?