News for hackers. Stuff that matters.
Re:Roblox October Outage Postmortem by wizwit999 on Thursday January 20, 2022 @05:37PM (#30016279)
> On October 27th at 14:00, one day before the outage, we enabled this feature on a backend service that is responsible for traffic routing. As part of this rollout, in order to prepare for the increased traffic we typically see at the end of the year, we also increased the number of nodes supporting traffic routing by 50%.

Seems like the smoking gun, this should have been identified and rolled back much earlier.

Re:Roblox October Outage Postmortem by Karrot_Kream on Thursday January 20, 2022 @08:51PM (#30018188)
If reading a postmortem makes the smoking gun obvious, then the postmortem is doing its job. Don't mistake the amount of investigation that goes into a postmortem for the available information and mental headspace during an outage.
Re:Roblox October Outage Postmortem by wizwit999 on Friday January 21, 2022 @03:03AM (#30020547)
I've been in my fair share of incidents so I'm aware of how they work. But they knew it was an issue related to Consul within hours. It shouldn't take more than two days before they check for recent deployments made to Consul.
Re:Roblox October Outage Postmortem by orangepenguin on Friday January 21, 2022 @02:21PM (#30028623)
In my experience, there's typically more than one "smoking gun". The problem isn't finding one, it's eliminating all of the "smoking guns" that aren't actually related to the outage.

If I worked at an organization with many teams deploying updates multiple times per day and several same day events seemed related, I would probably also put less weight on a gradual, months-long deployment that had completed a day prior.

Re:Roblox October Outage Postmortem by yuliyp on Thursday January 20, 2022 @08:37PM (#30018066)
It's obvious when it's pointed out in an article like this. It's less clear when it's one of many changes that could have been happening in a day, and it was an operation that was considered "safe" given that it had been done multiple times for other services in the preceding months.