News for hackers. Stuff that matters.
Re:Microsoft blames monster email outage on software defect ... by wvh on Tuesday November 17, 2026 @05:27AM (#50529661)
In my experience, we too often try to engineer around unavailability with caches, queues and cold serving, instead of just telling clients they can't have the cookie and have to wait. Maybe LBs should have a random drop policy in case of extreme overload of backend services, enough to keep things working somewhat both internally and visibly for clients, instead of contributing to the stampede and the whole system subsequently going completely down for all. "Too many" is a very real state.
Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @08:17AM (#50530959)
Google SRE Book - Handling Overload: https://sre.google/sre-book/handling-overload/

An important part is that clients pre-fail most requests and don't send them if the server is overloaded. They just assume the server would return an overload error.

Re:Microsoft blames monster email outage on software defect ... by trollbridge on Tuesday November 17, 2026 @08:48AM (#50531334)
This has been understated; at Google everything had to be engineered for a cold start. And they’d actual go test this periodically, using techniques to do so so end users never knew.