News for hackers. Stuff that matters.
Re:Roblox October Outage Postmortem by kreeben on Thursday January 20, 2022 @04:26PM (#30015289)
>> you need to load balance across multiple independent availability zones

The only problem with that is, there are no independent availability zones.

What we do have, though, is an architecture where errors propagate cross-zone until they can't propagate any further, because services can't take any more requests, because they froze, because they weren't designed for a split brain scenario, and then, half the internet goes down.

Re:Roblox October Outage Postmortem by outworlder on Thursday January 20, 2022 @04:29PM (#30015342)
> The only problem with that is, there are no independent availability zones.

There are - they can be as independent as you need them to be.

Errors won't necessarily propagate cross-zone. If they do, someone either screwed up, or they made a trade-off. Screwing up is easy, so you need to do chaos testing to make sure your system will survive as intended.

Re:Roblox October Outage Postmortem by kreeben on Thursday January 20, 2022 @04:36PM (#30015421)
I'm not talking about my global app. I'm talking about the system I deploy to, the actual plumbing, and how a huge turd in a western toilet causes east's sewerage system to over-flow.