News for hackers. Stuff that matters.
Re:Incident Report for Confluence by tyingq on Monday April 11, 2022 @08:33AM (#30987811)
It sounds to me like they lost some sort of master reference table that maps customers to services, instances, data, and so on. Which would mean a lot of manual examination to recreate the map. And would give you that situation where maybe the first thirds of restores is somewhat "fast" because it's the bigger customers with more data/logs/activity, and so easier to identify. But now they've hit the long tail.

They may also be gated by limitations in-between customers and the backups. Like backups going through a transit VPC, for example. You don't see the bandwidth limitation when you're sending incrementals all day, but try and do full, parallel, multiple customer restores through it, and ... (and perhaps they are afraid of making too many changes on top of the current chaos to fix something like that)

Re:Incident Report for Confluence by ollien on Monday April 11, 2022 @12:18PM (#30990927)
> It sounds to me like they lost some sort of master reference table that maps customers to services, instances, data, and so on.

What makes you say this?

Re:Incident Report for Confluence by tyingq on Monday April 11, 2022 @01:00PM (#30991670)
Just guessing what would drive a situation where some customers were restored quickly, others stuck for days, and matches somewhat with the limited statements they do make about the issue. Like this one: "The rebuild stage is particularly complex due to several steps that are required to validate sites and verify data."

Also matches with it being a cross-product outage.

But just a guess.

Re:Incident Report for Confluence by wlonkly on Monday April 11, 2022 @06:43PM (#30996074)
And not just a cross-product outage, but a cross-product outage that includes acquisitions. I doubt there is that much in common between the JIRA/Confluence tenancy model and, say, Statuspage.