Re:Microsoft blames monster email outage on software defect ...
by filleokus on Tuesday November 17, 2026 @08:03AM
(#50530808)
But given the size of the impact this incident had, I assume involving any team that would not be an issue.
I'm guessing there must be some deeper issue at play making it too risky or not possible. Maybe the hardware LB's were too intertwined with customer Azure loads or something?
But when you are hitting hour 12 of a global outage of one of your most central services I would assume even quite exotic solutions would be considered (e.g standing up dedicated hardware LB's with load shedding and pointing some of the DNS resolution to those)
Re:Microsoft blames monster email outage on software defect ...
by agentphil on Wednesday November 18, 2026 @12:51AM
(#50542349)
> The first is a smart DNS server that turns a lookup of mail.outlook.com or whatever into an IP address of a datacenter roughly nearby to where the user is, to a datacenter that's not too overloaded. But because DNS lookups are cached and sometimes cached too long by clients this is a very rough and ready form of LB.
This looks like a potential weak point against a cyber attack. A big number of distributed bots in one geographical area can send their requests to one data center and overload it.
Re:Microsoft blames monster email outage on software defect ...
by mike_hearn on Wednesday November 18, 2026 @03:59AM
(#50543740)
That's actually easy to solve. There was an attack exactly like that when I worked at Google (a loooong time ago). The DDoS bots cached the DNS lookup forever, but compliant clients respect the 5 min TTL. So the cluster was drained at the DNS level. User traffic went elsewhere and the botnet DDoSd an empty cluster nobody was using. Attack was defended against perfectly.
Most big cloud services operate several layers of load balancing. The first is a smart DNS server that turns a lookup of mail.outlook.com or whatever into an IP address of a datacenter roughly nearby to where the user is, to a datacenter that's not too overloaded. But because DNS lookups are cached and sometimes cached too long by clients this is a very rough and ready form of LB.
The second layer are routers that spread requests to a single IP address over "hardware" LBs that take raw TCP/IP connections from the internet. Then those machines do some basic health checking and forwarding to the frontend web servers. At that level the machines aren't aware of what service or customer the connections are for, they're just doing some flow hashing to pick a "backend" (another frontend).
Then the cloud/service level machines terminate the TCP/IP connection and often TLS as well. They're reverse proxies that do URL matching and forward traffic on to the app specific servers. This is the first layer at which you can start to intelligently inspect or drop traffic.
No one person in a big org has the ability to limit traffic to a service across every layer of load balancing that exists. If the reverse proxies start dying, reducing traffic would mean changing the configuration of the hardware LBs which is usually a different team.