TrogdorTheMan

Yup ^^ this. That and some dang help with managing security audits for our security team -.-. Just can't afford the man power to do it all.

From what I've seen on some internal email chains, there is a fail-over process for HA multi-region/site, but it didn't work right. Whomp whomp.

I'm willing to bet we can get at least the 20% return, they're still "officially" down. I bet it'll be a few hours till they say we're good.

Everyone make sure you've got detailed tickets into their service queue. Get your creds!

Oh lawd its starting to come back up! Pingdom monitor is flapping and we're starting to get some successful, albeit slow, logins!

I don't think so, I think that it was a combo of malicious intent and some indexes that never got run. I guess you might call it a bad migration since indexes didn't get run, but that seems more like a catalyst than a root. https://cdn.auth0.com/blog/20181128-Incident-RCA.pdf

heh, smells like its similar. thanks for the link.

Do you have a link to that previous post-mortem?

Yup, exactly. We're actively working on ripping them out of our software, this is ridiculous. I knew we should have taken the time to make a fail safe...sigh...

They're fast approaching the 95% availability mark for the month! Enterprise customers are already guaranteed 50% return on fees, I suspect they'll be giving out MUCH more than that.

We managed to get a reply from a C level. All we could get out of them was "something to do with our DB, but we don't know the root yet. Our fail-over process didn't work. This will never happen again".

Also, it only took them 2 and a half hours to admit it was their entire system instead of "a small subset of users" lol.