News for hackers. Stuff that matters.
Re:Microsoft blames monster email outage on software defect ... by k0morebi on Monday November 16, 2026 @04:33PM (#50523610)
Not a network engineer so genuinely curious how these things are handled at the epic scale of 365, azure, aws and so on.

Extent of my knowledge is nginx or traefik load balancing which at this scale is virtually equal to no knowledge. But seems like there would be systems in place to simply drop a sliding percentage of excessive traffic or block some regions once critical thresholds are reached rather than compounding the problem with forwarding the growing number of requests downstream to next victim server.

Re:Microsoft blames monster email outage on software defect ... by materielle on Monday November 16, 2026 @11:41PM (#50527526)
There’s all sorts of way to do it.

The easiest, and the default choice when I worked at Google, was exactly what you said: a sliding scale.

You could target racks, data centers, clusters, regions, etc. And “drain” the unit. For instance, stop routing traffic to cluster foo.

Then you could increment up stepwise 10%, 20%, etc.

Obviously we didn’t use nginx, and sure there’s interesting engineering involved. But it’s not like these systems are magic. There’s just a lot of clusters and a lot of load balancers. But at the end of the day, it’s just a load balancer.

As another Xoogler in this thread pointed out, there’s two problems:

1) Sometime teams accidentally introduce circular dependencies. Service foo might require service bar to boot and vice versa. Typically that’s fine, but from a rare cold start where both foo and bar are down, you’re in a pickle.

2) Load balancing is “easy”. But the services need to be designed with the aforementioned load balancing strategy in mind. Stuff gets tricky if there’s any pollers, process affinity, multiple components that need to process parts of the same user’s request in tandem. Given that outlook is ancient, I would not be shocked to learn that the system doesn’t respond well to naive “slowest ramp up traffic to a cluster” approaches. For big distributed applications, it takes a lot of meticulous design to build a system that responds well to simple SRE techniques like that.

That is one thing my team at Google would actually consider. We would make the code and architecture more complicated to accommodate the types of naive mitigations one would want to apply in an outage.

Re:Microsoft blames monster email outage on software defect ... by k0morebi on Tuesday November 17, 2026 @12:42AM (#50527865)
Thanks, I appreciate you taking the time to post the info.