It sounds to me like they lost some sort of master reference table that maps customers to services, instances, data, and so on. Which would mean a lot of manual examination to recreate the map. And would give you that situation where maybe the first thirds of restores is somewhat "fast" because it's the bigger customers with more data/logs/activity, and so easier to identify. But now they've hit the long tail.

They may also be gated by limitations in-between customers and the backups. Like backups going through a transit VPC, for example. You don't see the bandwidth limitation when you're sending incrementals all day, but try and do full, parallel, multiple customer restores through it, and ... (and perhaps they are afraid of making too many changes on top of the current chaos to fix something like that)

> It sounds to me like they lost some sort of master reference table that maps customers to services, instances, data, and so on.

What makes you say this?

Just guessing what would drive a situation where some customers were restored quickly, others stuck for days, and matches somewhat with the limited statements they do make about the issue. Like this one: "The rebuild stage is particularly complex due to several steps that are required to validate sites and verify data."

Also matches with it being a cross-product outage.

But just a guess.

And not just a cross-product outage, but a cross-product outage that includes acquisitions. I doubt there is that much in common between the JIRA/Confluence tenancy model and, say, Statuspage.

All I can say here is that my experience with ransomware response is that _every_ company has some document stating they can recover from scratch with backups in x hours, and in no case have I ever seen those hours be accurate, or even close by an order of magnitude.

Rebuilding the whole thing might end up being a lot easier than what Atlassians seems to be doing, which is restoring each user individually from a backup (?).

Though, going by their speed, this seems to be a weirdly manual process, not sure what is going on there.

> I can’t even imagine what is going on from a technical level.

I always wonder if most SaaS providers are fixated on building massive systems when simple shards would work for most customers. If you compare something like Gitea and GitHub you can probably see the reason. With Gitea your "shard" is siloed away from everyone else because there's no federation (yet?). GitHub is more like a huge social network where everyone can interact.

I think the vision for a lot of SaaS is to facilitate B2B collaboration by acting as a proprietary social network rather than via federation between relatively independent shards. The reason is simple IMO; vendor lock-in.

Former Gitea (Gogs) contributor here. I see your point and you're not really wrong but it's not that easy either:

- GitHub has way more features. More features means more scattered data (e.g. different micro-services databases, file storage).

- You cannot use Gitea in an organization with hundreds or thousands of engineers. It provides nether the features nor the scalability you need. I like Gitea but I'm happy that my company uses GitHub.

- Individuals and open-source use GitHub because it's free and has all the features (including stuff like free CI/CD which is expensive). This is all being paid for by companies like mine who buy expensive enterprise licenses for thousands of engineers.

- It's not uncommon to shard by customer/account, even in the cloud and GitHub is actually trying to do this right now for their primary database. Doesn't solve the problem with different storages for different micro-services though.

Chances are their code is older than yours, hence less automated, and they've since lost some of the people who would know how to do some of the most difficult manual things on failure.

I would guess outsourcing to contractors is involved in this mess somewhere.

It's also one of those companies built by acquisitions.

> If you blew up our entire AWS account save our backups we could be nominally back online within an hour or two and fully within 6.

Do you regularly exercise this, or is it just a plan?

We've brought entirely separate instances up a number of times, from scratch. We sell straight to governments and some of them have very strict hosting requirements.

Our Chinese installation for instance runs on servers controlled by the Chinese government. I believe they run an AWS compatibility layer from what I've heard, I haven't personally worked on this.

We've got a decent amount of experience bringing the whole thing up from scratch, and we build with that in mind. Most of the time in my estimate is for importing the larger detail tables, which we do regularly for our staging environment.