Re:Roblox October Outage Postmortem
by outworlder on Thursday January 20, 2022 @04:29PM
(#30015342)
> The only problem with that is, there are no independent availability zones.
There are - they can be as independent as you need them to be.
Errors won't necessarily propagate cross-zone. If they do, someone either screwed up, or they made a trade-off. Screwing up is easy, so you need to do chaos testing to make sure your system will survive as intended.
The only problem with that is, there are no independent availability zones.
What we do have, though, is an architecture where errors propagate cross-zone until they can't propagate any further, because services can't take any more requests, because they froze, because they weren't designed for a split brain scenario, and then, half the internet goes down.