But it wasn't a hardware issue. It was a software one and that would have crossed AZ boundaries.
So then why does the post mortem suggest setting up multi-az to address the problems they encountered?
I took that to mean sharding Roblox instead of spanning it across data center AZs.
FTA:
> Running all Roblox backend services on one Consul cluster left us exposed to an outage of this nature. We have already built out the servers and networking for an additional, geographically distinct data center that will host our backend services. We have efforts underway to move to multiple availability zones within these data centers; we have made major modifications to our engineering roadmap and our staffing plans in order to accelerate these efforts.
If they were in AWS they could have used Consul across multi-AZs and done changes in a roll out fashion.
So that next time they can spend 96 hours on recovery, this time adding a split brain issue to the list of problems to deal with. Jokes aside, the write-up is quite good after after thinking about all the problems they had to deal with, I was quite humbled.
It doesn't really explain how they reached the conclusion that that would help. Like, yes, it's a problem that they had a giant Consul cluster that was a SPOF, but you can run multiple smaller Consul clusters within a single AZ if you want.
Honestly it reads to me like an internal movement for a multi-AZ deployment successfully used this crisis as an opportunity.
This is exactly one of the reasons people go cloud. Introducing an additional AZ is a click of a button and some relatively trivial infrastructure as code scripting, even at this scale.
Running your own data center and AZ on the other hand requires a very tight relationship with your data center provider at global scale.
For a platform like Roblox where downtime equals money loss (i.e. every hour of the day people make purchases), then there is a real tangible benefit to using something like AWS. 72 hours downtime is A LOT, and we're talking potentially millions of dollars of real value lost and millions of potential in brand value lost. I'm not saying definitively they would save money (in this case profit impact) by going to AWS, but there is definitely a story to be had here.