News for hackers. Stuff that matters.
Re:Microsoft blames monster email outage on software defect ... by andix on Monday November 16, 2026 @07:48PM (#50525888)
There is one thing I don't get with this issue. The problem was basically that their system couldn't handle the normal load. Why couldn't they just reduce the load at some point just by dropping/limiting connections?

For example just accept connections from specific (random) IP ranges. They could even cycle through different ranges every 5 minutes to give clients a chance to sync at least once every hour.

The reduced load could be used to gradually bring up servers again without failing, add additional capacity, and gradually restoring access. Completely without fixing the bug at all. The system managed to handle usual load even with the bug, it just hit some critical point that made it collapse. Adding more resources would also fix that.

Re:Microsoft blames monster email outage on software defect ... by itopaloglu83 on Monday November 16, 2026 @08:37PM (#50526283)
You mean they should've made disaster recovery plans ahead of time? That requires experience and a certain level of know-how, in a company that has just decided to "voluntarily retire” a quarter of its most experienced workforce. There was a lot of things they could've, should've, and would've done.
Re:Microsoft blames monster email outage on software defect ... by andix on Monday November 16, 2026 @08:56PM (#50526419)
I came up with another theory: maybe it was internal load, server to server communication. For example email traffic between Outlook mailboxes (or some kind of API calls). Once the queues were filled up, every server added back to the pool just increased internal load by hammering other servers.

Restricting external access wouldn't solve this issue.

I guess adding a config option for throttling queue processing is something any distributed system should have.

Re:Microsoft blames monster email outage on software defect ... by mike_hearn on Tuesday November 17, 2026 @05:05AM (#50529521)
The issue appears to have been in one of the load balancer layers themselves (the "front door"). So that makes it a lot harder.

Most big cloud services operate several layers of load balancing. The first is a smart DNS server that turns a lookup of mail.outlook.com or whatever into an IP address of a datacenter roughly nearby to where the user is, to a datacenter that's not too overloaded. But because DNS lookups are cached and sometimes cached too long by clients this is a very rough and ready form of LB.

The second layer are routers that spread requests to a single IP address over "hardware" LBs that take raw TCP/IP connections from the internet. Then those machines do some basic health checking and forwarding to the frontend web servers. At that level the machines aren't aware of what service or customer the connections are for, they're just doing some flow hashing to pick a "backend" (another frontend).

Then the cloud/service level machines terminate the TCP/IP connection and often TLS as well. They're reverse proxies that do URL matching and forward traffic on to the app specific servers. This is the first layer at which you can start to intelligently inspect or drop traffic.

No one person in a big org has the ability to limit traffic to a service across every layer of load balancing that exists. If the reverse proxies start dying, reducing traffic would mean changing the configuration of the hardware LBs which is usually a different team.

Re:Microsoft blames monster email outage on software defect ... by filleokus on Tuesday November 17, 2026 @08:03AM (#50530808)
But given the size of the impact this incident had, I assume involving any team that would not be an issue.

I'm guessing there must be some deeper issue at play making it too risky or not possible. Maybe the hardware LB's were too intertwined with customer Azure loads or something?

But when you are hitting hour 12 of a global outage of one of your most central services I would assume even quite exotic solutions would be considered (e.g standing up dedicated hardware LB's with load shedding and pointing some of the DNS resolution to those)

Re:Microsoft blames monster email outage on software defect ... by agentphil on Wednesday November 18, 2026 @12:51AM (#50542349)
> The first is a smart DNS server that turns a lookup of mail.outlook.com or whatever into an IP address of a datacenter roughly nearby to where the user is, to a datacenter that's not too overloaded. But because DNS lookups are cached and sometimes cached too long by clients this is a very rough and ready form of LB.

This looks like a potential weak point against a cyber attack. A big number of distributed bots in one geographical area can send their requests to one data center and overload it.

Re:Microsoft blames monster email outage on software defect ... by mike_hearn on Wednesday November 18, 2026 @03:59AM (#50543740)
That's actually easy to solve. There was an attack exactly like that when I worked at Google (a loooong time ago). The DDoS bots cached the DNS lookup forever, but compliant clients respect the 5 min TTL. So the cluster was drained at the DNS level. User traffic went elsewhere and the botnet DDoSd an empty cluster nobody was using. Attack was defended against perfectly.
Re:Microsoft blames monster email outage on software defect ... by skywhopper on Tuesday November 17, 2026 @09:16AM (#50531691)
The system responsible for reducing the load was the system that was failing.