News for hackers. Stuff that matters.
Re:Microsoft blames monster email outage on software defect ... by ZiiS on Monday November 16, 2026 @10:17AM (#50518488)
- we added a bug (yes, everyone does).

- our testing didn't cover it (again, this is expected occasionally).

- this cascaded to take down more than a billion people's service (WTF; this is not how you design systems).

- it took 15 hours to deploy the fix (not good, but at this scale maybe).

Feels like they are completely missing the point of where the problem is and how to fix it?

Re:Microsoft blames monster email outage on software defect ... by lokar on Monday November 16, 2026 @10:20AM (#50518531)
In terms of cascading failure and inability to rollback or fix quickly, MS has always been terrible.
Re:Microsoft blames monster email outage on software defect ... by bayindirh on Monday November 16, 2026 @10:36AM (#50518712)
Also, don't forget that e-mail is a distributed system and servers retrying always keep a pressure on you, despite their best intentions, and spaced retries.

So, even when you fix the issue, it takes a bit to clear up, and the load gauge stays at the red zone for a while.

Re:Microsoft blames monster email outage on software defect ... by lokar on Monday November 16, 2026 @10:45AM (#50518835)
Anytime you run a proxy system for a large number of clients you don’t control you need a way to cold start. This means being able to allow traffic through (cheaply!) on a client basis, rejecting the rest.

This lets you start at a very small amount of load and gradually ramp up. But you can’t reject randomly or goodput stays at zero and the backlog never decreases.

Re:Microsoft blames monster email outage on software defect ... by bayindirh on Monday November 16, 2026 @10:46AM (#50518859)
Sounds reasonable.

I believe SDN systems can do that with ease, but what would be the equivalent method on older systems? Selectively allowing IPs to go through the trickle load into the system?

Re:Microsoft blames monster email outage on software defect ... by lokar on Monday November 16, 2026 @10:49AM (#50518906)
IP is often hard on the proxy/load balancer, you get a really large list, which becomes expensive.
Re:Microsoft blames monster email outage on software defect ... by bayindirh on Monday November 16, 2026 @10:51AM (#50518920)
This is also what I thought, so why I asked, actually.
Re:Microsoft blames monster email outage on software defect ... by toast0 on Monday November 16, 2026 @07:11PM (#50525515)
Selectively allowing IPs is a reasonable way to do it. If you think about it, it's terrible, but it will get you going, and it's pretty easy do something like start with allow nothing, then allow all ips under 2.0.0.0, then 3.0.0.0, etc.

Distribution of users by /8 isn't very even and the order that customers will get online wouldn't be an order anyone would pick, but it should be reasonable to implement and synchronize (if needed) and it lets you make progress.

Re:Microsoft blames monster email outage on software defect ... by DenisM on Tuesday November 17, 2026 @01:04PM (#50535518)
I’m wondering if you’re preparing for cold start ahead of time you could create a better cold start map that’s distributed more evenly for more predictable turn up, and also with better affinity so you can you report progress in a way people can understand ( eg state by state). It would be a pain to load and activate the map into edge routers in a hurry, but maybe be necessary. Some superscalers are deep into writing their own networking firmware, this could go into design requirements for text revision.

Another idea is to have different clients use different ports (enterprise customers on one port, SMB on another) and turn up in the order of SLA.

Re:Microsoft blames monster email outage on software defect ... by nijave on Monday November 16, 2026 @08:21PM (#50526176)
They could drop traffic at L3 before it even hits their servers.
Re:Microsoft blames monster email outage on software defect ... by lokar on Tuesday November 17, 2026 @11:07AM (#50533387)
That's what we are talking about. But you need a way to manage that L3 traffic by client.
Re:Microsoft blames monster email outage on software defect ... by VirusNewbie on Monday November 16, 2026 @05:53PM (#50524677)
>- it took 15 hours to deploy the fix (not good, but at this scale maybe).

I've definitely been in similar scale war rooms and 15 hours is not acceptable.

Re:Microsoft blames monster email outage on software defect ... by mrheosuper on Monday November 16, 2026 @09:36PM (#50526707)
Can you share more, it's not like we have billions-user affected bug everymonth.
Re:Microsoft blames monster email outage on software defect ... by VirusNewbie on Monday November 16, 2026 @11:20PM (#50527396)
sure, it's not that longer outages can't happen, but it's very peculiar that the outage was global, and that mitigation took that long. I would have expected fast mitigation for the most populous zones or regions.

The thing with plain outages vs. say, partial degradation of services, you can roll out fast because you're already at worst case scenario *[1].

Here's an interesting one to compare to that I'm very familiar with:

https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbn...

Another really bad one I did not have any involvement in but studied extensively post hoc:

https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

*[1] - I suppose data loss is worse but let's assume most services are stateless.

Re:Microsoft blames monster email outage on software defect ... by tremon on Tuesday November 17, 2026 @12:27PM (#50534928)
What do you mean, "not acceptable"? Given the situation, I think 15 hours is defensible. Where do you suggest putting the blame for the implementation and deployment of the fix?

What's not acceptable to me is a bug of this impact going undetected for 6+ months in production.

Re:Microsoft blames monster email outage on software defect ... by topaz0 on Monday November 16, 2026 @06:32PM (#50525131)
Antitrust
Re:Microsoft blames monster email outage on software defect ... by nijave on Monday November 16, 2026 @08:20PM (#50526164)
The irony is they had a more fault tolerant system 30 years ago with on prem Exchange. I'm not sure why they decided to make it a global mega service. Some things are a bit more justifiable with shared state requirements but email is stupidly distributed and fault tolerant.

You can't even say "but managing fleet/cells is harder": they make software for that, too!