Re:Microsoft blames monster email outage on software defect ...
by ncr100 on Monday November 16, 2026 @09:29AM
(#50517969)
They talk about their relatively low-end testing environment PASSing the July code, and being different from their high-end production environment machines, sadly hiding the concurrency bug during the test phase.
Is this commonly acceptable? I'd imagine it's a costly trade-off?
Re:Microsoft blames monster email outage on software defect ...
by lokar on Monday November 16, 2026 @10:18AM
(#50518509)
I always this was standard to have a test setup in the production environment on the latest hardware (that’s how you qualified that hardware in the first place) running one “unit” of the system (whatever that is) at full scale and load (ideally higher).
Re:Microsoft blames monster email outage on software defect ...
by jakewins on Monday November 16, 2026 @04:28PM
(#50523546)
We had this conundrum early on when we were building the Neo4j DBaaS - there was zero chance to afford to run test environments as big as production.
We solved that by sharding customers, so about every 1000 database clusters got their own k8s cluster; production then could run hundreds of those “orchestras”, but we could afford to load test one of them and be relatively confident the test was valid for production.
This relied on the various cloud vendors running the front-line load balancers that took traffic and moved it into the shards though. I was about to say that seemed to be the case for Microsoft here as well, but actually it seems they load tested an individual server with different specs than prod.
Tricky problem though - it seems they reasoned if a lower specced server passed the load test, a much beefier prod server would handle the load too. Wrong, but you can see why someone made that assumption.
Re:Microsoft blames monster email outage on software defect ...
by aspbee555 on Monday November 16, 2026 @07:14PM
(#50525549)
I have found that AI really loves using locking/Mutex when working with rust and I have to specifically tell it to not use any locking/use atomics/swaps instead
Re:Microsoft blames monster email outage on software defect ...
by nl on Monday November 16, 2026 @11:11PM
(#50527331)
> So, it's more the former. The testing machine is probably a 8 year old server on an office shelf
I doubt this very much.
The load testing env is the Office 365 load testing environment. I'd be surprised if it is less than 10,000 cores, and the machines are probably good but older; possibly 50 core Xeons instead of 128 core EPYC or something.
It seems likely that the test env was setup to mirror one of the smaller 365 regions a few years ago and the only updates were when machines were replaced.
Re:Microsoft blames monster email outage on software defect ...
by cowlevel on Tuesday November 17, 2026 @08:36AM
(#50531169)
Mutices are hard, concurrency is hard, we don't notice the deficiencies in the Linux stack as much for some reason. Nobody's explained why my GUI is laggy while running a crypto miner (for warmth) at nice level 20.
Re:Microsoft blames monster email outage on software defect ...
by malkia on Tuesday November 17, 2026 @12:16PM
(#50534705)
I believe this must be NUMA cache line contention. If CI testing involved a VM, the VM was most likely cordoned to run in single VM node, or at least not crossing (because it does not make sense otherwise), unless it was special VM to test this case.
I've seen NUMA cache line contention affecting us for real (on Windows, for our game tools). Limiting certain tools to run locked under one NUMA node helped a lot. It's real.
Re:Microsoft blames monster email outage on software defect ...
by notyourwork on Monday November 16, 2026 @11:02AM
(#50519087)
Testing at production scale is extremely difficult. It’s also financially expensive. In my experience, you balance the two where big changes are tested with some simulation setup to gain confidence. Though, the ci/cd pipeline never has close to full scale automated testing.
Re:Microsoft blames monster email outage on software defect ...
by Choad_20261104 on Monday November 16, 2026 @11:28AM
(#50519438)
...and many times you test to experience, and experience cannot possibly build interferences that simulate the instances you have not experienced nor what has been imagined based on that experience.
some ways of precipitating the demons that are in the ontology of not knowing what you don't know are monte carlo, or perhaps noise injection, but those are time consuming, quite wasteful, and truly hated by people used to working with deterministic machines. that makes finding corner cases very difficult
plus, mistakes are the essence of learning and sometimes mistakes are necessary.
likewise, we are a very efficient world and efficiency hates redundancy, and hates nonconformity...of which those are two of the features that make exploits widely successful and failures massively impactful
Re:Microsoft blames monster email outage on software defect ...
by hinkley on Tuesday November 17, 2026 @05:50AM
(#50529808)
> efficiency hates redundancy
We need to teach queuing theory in school. Intuition about how much redundancy you can squeeze out if a system and still handle hiccups is woefully inaccurate. So we under provision and are surprised when traffic jams take hours and hours to clear. Despite that being completely predictable.
Re:Microsoft blames monster email outage on software defect ...
by cowlevel on Tuesday November 17, 2026 @05:57AM
(#50529849)
Basic intuition: if you handle 900 users per minute with a capacity of 1000, and then freeze for one minute or just get an above average usage burst, it'll take 9 minutes to use that excess capacity of 100 to serve that backlog of 900.
If you get 990 with a capacity of 1000, it takes 99 minutes to clear that same disruption. And if you get 1000 with a capacity of 1000, average time to recover from a little random fluctuation is infinity.
I believe normal practice is to upgrade fiber-optic lines when they're somewhere between 50% and 80% utilisation.
Re:Microsoft blames monster email outage on software defect ...
by hinkley on Tuesday November 17, 2026 @11:57AM
(#50534380)
This is the “common sense intuition isn’t” situation. Most people don’t even get this far, as is evidenced by their surprise at why things take so long to get back to normal.
Also “9 minutes to clear” only tells how long the congestion is, NOT what the customer experiences. On a typical response time of .6s, after the delay the first customer sees 101x the usual response time tapering off to 2x almost 9 minutes later. That is a very different story than, “things will be slow for 9 minutes”. It’s everything will be an absolute shitshow for 9 minutes.
Re:Microsoft blames monster email outage on software defect ...
by cowlevel on Tuesday November 17, 2026 @09:48PM
(#50541237)
Related: if a network link is at 90% utilisation, then on average each packet will be stuck behind 9 other packets, in a simplistic model. There's a 90% chance the link is already occupied when the new packet arrives, an 81% chance that slot and the next one are both already occupied, a 72.9% chance that slot and the next two are already occupied, etc. the average delay is 1/(1-r) which is 10 at 90%, 100 at 99%, etc. this works for any type of queue, such as a line to buy movie tickets.
Re:Microsoft blames monster email outage on software defect ...
by Choad_20261104 on Tuesday November 17, 2026 @02:49PM
(#50537145)
school probably does teach it, students likely don't absorb it. Mine came in a Fluid Dynamics course believe it or not.
Professor made a simple statement, pointed to the local (I405) freeway in Irvine and stated that traffic is incompressible flow similar to water. Flow problems are only solved by a bigger pipe and whenever you see its manifestations (flow memory, slowing at chokepoints, etc.) remember that fact. Also remember: there will always be a single slowest point in a network unless you can somehow manage to cause the flow to accelerate past and away from obstructions.
Re:Microsoft blames monster email outage on software defect ...
by hinkley on Tuesday November 17, 2026 @07:19PM
(#50540196)
Mine didn't, but I got a few things I expected everyone else to learn that nobody has, like distributed computing.
The first guy I knew who was write-a-book invested in Agile was working out Fluid Dynamics analogies with me when we got talking shop. We stopped working together before I made the queuing theory connection.
Re:Microsoft blames monster email outage on software defect ...
by jamesfinlayson on Monday November 16, 2026 @09:23PM
(#50526626)
Yeah, my day job's peak is at the very extreme, thousands of users a second. I believe there is a dedicated set of servers to load test this scenario, which while not cheap, is not cost prohibitive, and it's deemed absolutely critical that the system be able to handle this, so the business is happy to foot the bill.
Re:Microsoft blames monster email outage on software defect ...
by Anon1096 on Tuesday November 17, 2026 @05:38AM
(#50529721)
Here we're talking far more than a million requests per second. You can't buy the spare hundreds of thousands to millions of servers needed to do a separate non production load test of something at this scale no matter how much money you are willing to spend.
Re:Microsoft blames monster email outage on software defect ...
by munchbunny on Monday November 16, 2026 @05:19PM
(#50524229)
At Microsoft's scale, it's extremely difficult to simulate production scale because the sheer size of the production system is overwhelming, and many issues emerge from the combination of sheer volume and the complexity that reality tends to introduce.
Microsoft tests its stuff on their own employees (old citation, but still true today: https://devblogs.microsoft.com/oldnewthing/20110802-00/?p=10...), and that includes Exchange online. However, even though Microsoft is a pretty big company, it's still teeny compared to its customer base.
Re:Microsoft blames monster email outage on software defect ...
by hinkley on Tuesday November 17, 2026 @05:53AM
(#50529824)
Some projects end up slowing rolling restarts down, in the hopes that alarms will start to fire if the new servers are not keeping up like the older version did. Then someone forgets why it’s that way and we are quickly reminded and it gets restored to the old timing.
Re:Microsoft blames monster email outage on software defect ...
by awesome_dude on Monday November 16, 2026 @06:18PM
(#50524981)
I read that part of their statement as "QA were set a priority to ensure existing hardware wasn't affected, which reduced their (QAs) capacity to test on new hardware which was assumed to be safe"
They find QA dirty in their statement, because they don't state who set that priority, nor why the new hardware was felt to be less likely to be affected (what did their risk matrix look like for their hardware profiles i wonder)
Re:Microsoft blames monster email outage on software defect ...
by elcritch on Monday November 16, 2026 @10:29AM
(#50518633)
It's a rhetorical jab of course.
The mantra of "shareholder primacy" sounds great in theory but just becomes an excuse for short term profit boosts for a small class of shareholders usually at the expensive of the long term health and growth of the company and long term value seeking shareholders.
Sort of similar to how most countries with "people's" or "democratic" in the government's name tend to be democratic only for the ruling party.
In this case ensuring adequate engineering resources for possibly the most widely used email system in the world doesn't seem crazy. How many long term customers will they loose because of this incident? Perhaps they did have enough resources, but given the recent layoffs to chase AI I sort of doubt it.
Re:Microsoft blames monster email outage on software defect ...
by rs999gti on Monday November 16, 2026 @01:24PM
(#50520947)
> It's a rhetorical jab of course.
Not for me and probably a lot of people and institutions too. We have retirement accounts, endowments, and pensions tied to index funds and the index funds have these stocks in them.
We need these companies to do well to ensure we retire and grow well.
Re:Microsoft blames monster email outage on software defect ...
by rs999gti on Monday November 16, 2026 @01:25PM
(#50520962)
False. A lot of working people, probably even you, are tangentally invested into these stocks through indexes, pensions, endowments, and retirement accounts.
Re:Microsoft blames monster email outage on software defect ...
by triceratops on Monday November 16, 2026 @02:00PM
(#50521454)
What's false about it? I didn't say working people don't own stocks. They just don't own most of them. Half of the benefit of any increase in stock prices goes to 1% of the people.
Workers benefit much more from higher wages than higher stock prices. Simply because much more of their income comes from wages. So whenever someone says "Think about the retirees and other ordinary shareholders" in response to a suggestion to pay workers more than absolutely necessary, they're full of shit.
Re:Microsoft blames monster email outage on software defect ...
by qwerpy on Monday November 16, 2026 @03:43PM
(#50522972)
I don't care who owns most of them, I care that a big part of my retirement savings are in stocks. It may be an inconvenient fact for some people, but my interests are aligned with "the rich" for this particular thing. If some rich guys get richer when MSFT does well, but I also get to have a more comfortable retirement as well, then that's fine by me.
This "rich people will benefit so it's automatically bad" mentality is counterproductive and borne out of jealousy.
Re:Microsoft blames monster email outage on software defect ...
by triceratops on Monday November 16, 2026 @04:20PM
(#50523461)
Your interests are only aligned when you're actually retired and 100% of your income comes from stock investments.
If part of your retirement income comes from another source, such as rental property, workers being paid more is better for you because they can afford higher rents.
Ordinary workers would better off if they were simply paid more and could put more money into their retirement savings.
Re:Microsoft blames monster email outage on software defect ...
by rs999gti on Monday November 16, 2026 @04:52PM
(#50523860)
This is HN, 99% are white collar workers like me invested into pensions, investment accounts, funds, etc. that are mostly stock backed due to working at companies with these programs. Even government and non-profits are the same.
We all need to side with whoever makes the charts go up and to right to ensure our post-retirement lives are well funded.
Myself and many like me don't care what side that is and we don't care much for policies, groups, or programs that are counter to that.
Re:Microsoft blames monster email outage on software defect ...
by triceratops on Monday November 16, 2026 @05:04PM
(#50524030)
My guy what's going to make your retirement account go up faster? Contributions or gains?
For at least the first 11 years of most people's careers, contributions outweigh gains. And only if they continuously contribute the same amount every month.
Being paid more, especially early career, is just more useful to the 99% white collar workers on HN. It lets you hit the crossover point faster. It lets you save more money for retirement. Compounding more money earlier gives you a better retirement. This is simple math. It isn't a class conflict argument.
Re:Microsoft blames monster email outage on software defect ...
by qwerpy on Monday November 16, 2026 @05:21PM
(#50524241)
Did I miss something? I don't see where someone in this comment chain argued for workers to not be paid more. I very much enjoyed being paid well so that I could retire early! It would be great if others get to be similarly fortunate.
All me and the other guy are saying is that it's possible to not be filthy rich and still want stocks to do well. "Think of the poor starving shareholders" implies that we commoners shouldn't care or want that. But that's false. I'm no longer working, and have no rental income, so I need my investments to do well.
Re:Microsoft blames monster email outage on software defect ...
by LtWorf on Tuesday November 17, 2026 @08:41AM
(#50531236)
> This "rich people will benefit so it's automatically bad" mentality is counterproductive and borne out of jealousy.
The fact that you can't apply logical reasoning is your own limitation and you shouldn't presume it applies broadly to everyone.
For example in my country the previous extreme right wing government lowered taxes to high earners. That includes me. I did pay slightly less in taxes, while people who make much more money than me had a much bigger reduction.
So I also had an advantage? LOL NO.
I did not because they also doubled the cost to go and see a doctor so my slight tax reduction wasn't even enough to pay for that.
Of course people who can't do math think "uh good less tax! I vote for them again!", but people who can do some 3rd grade mathematics might realise that they actually paid more.
And we aren't even including the increased health risk in going around when the less wealthy people will naturally tend to go less to the doctor because they are being priced out and infect everyone.
Re:Microsoft blames monster email outage on software defect ...
by rs999gti on Monday November 16, 2026 @04:58PM
(#50523946)
Who cares? For most working people like myself invested into these companies through funds, pensions, investment accounts, savings, etc., all we care about is up and to the right forever or at least when we come into retirement.
Re:Microsoft blames monster email outage on software defect ...
by dgrin91 on Monday November 16, 2026 @09:55AM
(#50518239)
Doesn't the article state the opposite? The low end was not the problem. It was high end that was the problem.
I like ragging on Microsoft as much as the next guy, but honestly this problem sounds complex & hard to predict. AI or not, this feels like the type of bug that would have slipped by me as well.
The response clearly could have been stronger though.
Re:Microsoft blames monster email outage on software defect ...
by deeringc on Monday November 16, 2026 @05:17PM
(#50524199)
Seems like the issue happens readily at high load on a system with many cores. Once the conditions were right it seemed to impact a large proportion of their fleet. They just didnt have that combination in their test/validation pipeline. They likely have some sort of load test in their pipeline, but it's running on machine that's much smaller than their production machines... so it didn't trigger the lock contention bug.
The opposite is true running in production for the last several months. They were runnign on big production machines with enough cores, but never quite had enough load to trigger the lock contention. But once both conditions were true, big machines and high load, due to some unrelated failover elsewhere suddently bringing more load, we then had the magic combination.
It's really not unreasonable to expect an enormous hyperscaler like Microsoft to do load testing on hardware that exactly matches their production systems.
Re:Microsoft blames monster email outage on software defect ...
by zdragnar on Monday November 16, 2026 @10:09AM
(#50518418)
I read "more than one low end" as "more testing environments than one low end" rather than "more low end environments than one". It's a snarky way of saying they should have been testing high end environments for no more reason than they had the money to do it, rather than saying they should have been testing the low end more.
Re:Microsoft blames monster email outage on software defect ...
by preommr on Monday November 16, 2026 @09:57AM
(#50518270)
This is what happens when you burn billions for no reason.
Github copilot had INSANE usage. Cursor switched away from per request plans in 2025, Copilot did it this year with all the reasoning and more expensive models. It was insane, you could write out a whole page of features, and have it run on one request for hours depending on the loopholes you exploited (some models were discounted for some time, basically free in some cases, and shenanigans around switching in betwen agents)
We talk about opus et. al being heavily subsidized at like 1-2k for $30-$100. I remember seeing figures in the 10s of thousands in subsidizations for GHC. There's no way they weren't losing just massive piles of money.
Re:Microsoft blames monster email outage on software defect ...
by hmokiguess on Monday November 16, 2026 @09:52AM
(#50518209)
> The engineers who worked feverishly to restore services deserve commendation and will hopefully never have to buy their own beverages in Redmond again (or at least escape the next round of firings).
Re:Microsoft blames monster email outage on software defect ...
by bayindirh on Monday November 16, 2026 @09:56AM
(#50518252)
In sysadmin speak this is hoping that Microsoft gives them free beverage credits (or give them some kind of bonus because of their efforts) and don't fire them as soon as the fix is deployed to scapegoat them.
In a rare occurrence, The Register sides with the Microsoft developers. Not that they're anti-developers, but they are anti Microsoft because of its well-earned reputation for EEE and Halloween Documents.
Re:Microsoft blames monster email outage on software defect ...
by deathanatos on Monday November 16, 2026 @05:13PM
(#50524149)
Nurses reported many hospitals paid significant bonuses and hazard pay during COVID, including (already mandated) time-and-a-half, as nurses were in short supply. Many nurses were being paid at hourly rates above that of senior software engineers.
That's not to say their job was or is easy: nursing is far more hazardous than SWE, and significantly more physically and mentally demanding, too. But I do think many got more compensation than your $5 gift certificate implies.
But as a software engineer, if I work overtime on an outage like the above, I'll get precisely $0 of additional pay for it.
Re:Microsoft blames monster email outage on software defect ...
by jamesfinlayson on Monday November 16, 2026 @09:25PM
(#50526632)
Yeah I had a family member working a nurse at the time. Not sure how much she was making but it sounded pretty lucrative and she was definitely in demand.
Re:Microsoft blames monster email outage on software defect ...
by cowlevel on Tuesday November 17, 2026 @08:38AM
(#50531194)
Every one I've ever seen - in countries with actual labour rights - would give you either the same credit in time off, that you'd usually take on the next workday, or double that.
Re:Microsoft blames monster email outage on software defect ...
by voidfunc on Monday November 16, 2026 @10:27AM
(#50518612)
It means they deserve "Thanks" (buying a beer/coffee is a token Thank You) and hopefully they are removed from the pool of candidates for the next layoff.
Surprised so many people are struggling with this...
Having been on incidents like this before though, yea, that aint happening.
Re:Microsoft blames monster email outage on software defect ...
by 27183 on Monday November 16, 2026 @10:32AM
(#50518663)
> Having been on incidents like this before though, yea, that aint happening.
Yeah, instead what will happen is their manager will ask them why they didn't get as many story points done this sprint as they did last sprint and their performance will be marked "degraded".
Re:Microsoft blames monster email outage on software defect ...
by bayindirh on Monday November 16, 2026 @10:32AM
(#50518675)
> Having been on incidents like this before though, yea, that aint happening.
Been there, through that, and sadly you're right. The bitter part is I didn't want extra compensation or anything. Just a thank you and a day off is more than enough for me, so to recharge and ride the wave of what we (not I, we) have accomplished, so I can work with extra motivation for some time.
Instead, got a 5 minute coffee break and dived to the next non-urgent ticket. Yeah, great way to thank and motivate the people who spent 24 hours in an underground office cutoff from the world, trying to save a system from melting down.
How long until Microsoft fires key people to the point that outages become more common? All in the name of AI, which I don't believe for a second, they're just trying to maintain stock market profitability math by firing enough, they'll just hire more people when they make more.
Re:Microsoft blames monster email outage on software defect ...
by thewebguyd on Monday November 16, 2026 @11:15AM
(#50519256)
> How long until Microsoft fires key people to the point that outages become more common?
Already happening. Granted, not a global outage like this one, but there's not a single day without a service degradation or advisory on the admin portal for 365. The "It's now called Microsoft 360" memes are well earned, and their support is atrocious. I get the feeling that everyone who knows how it works is gone.
Re:Microsoft blames monster email outage on software defect ...
by Choad_20261104 on Monday November 16, 2026 @11:18AM
(#50519277)
The opportunity/command to exit is not always negative. In many of my career moves, it turned out that the next job was better and came far sooner than the severance calculation had allowed for. Some within weeks. Hardly had time to relax. It seems like for talented individuals opportunity always pops up.
Re:Microsoft blames monster email outage on software defect ...
by Choad_20261104 on Tuesday November 17, 2026 @02:43PM
(#50537079)
fair enough. I have been there and I used those as opportunities to observe the results. MOST times the results were actually positive for the company and the remaining staff, and when I was one of them it was definitely good.
SOMETIMES the company was being prepared for sale...THEN you can only hope you get let go in the first few rounds BEFORE the money for severances runs out
I am sorry that somehow society is beginning to sell the idea that the American workforce does not or need not include rough and ready individuals.
I am aged out of the workforce at this point and find it hard to re-enter due to different problems
Re:Microsoft blames monster email outage on software defect ...
by sandworm101 on Monday November 16, 2026 @09:57AM
(#50518273)
>> "an update that returned the code path to the previously known good state in Windows."
I know I shouldn't be surprised, but I still struggle to imagine such high-capacity servers running with windows. Part of me wonders why any important server would be running windows. And I do chuckle at the "update" that returned windows to a previous state. Never admit a rollback. Never admit a previous update was less than perfect. Hold the line MS. Hold that line.
Re:Microsoft blames monster email outage on software defect ...
by thesuitonym on Monday November 16, 2026 @10:09AM
(#50518411)
Yeah it's pretty crazy that Microsoft, a company most well known for it's Windows operating system, would use that same operating system on its servers.
Re:Microsoft blames monster email outage on software defect ...
by stackskipton on Monday November 16, 2026 @10:14AM
(#50518463)
Alot of 365 is using Exchange/Sharepoint/Skype for Business code bases which were Windows specific. With Exchange, there was specifically role called "Client Access" which clients connect, the system looks up which backend server has your mailbox and proxies your connection there. Reading between the lines, my guess is this system was reused for Sharepoint/Teams/Office and they could swing off of it to secondaries but Exchange is hard locked on it.
Microsoft does greenfield new stuff on Linux mostly.
Re:Microsoft blames monster email outage on software defect ...
by theragra on Monday November 16, 2026 @10:54AM
(#50518958)
But if they wouldn't do it, Windows will be even worse.
Moreover, my current switch to Linux shows that Linux is not so much better in stability as it is in public imagination. Once a month or two, I have KDE freeze so hard I barely can recover without reset (sometimes can't). Still can't find the reason. Hardware seems to be okay, it is an expensive gaming laptop.
Also, out of RAM can lead to GUI crashing even with swap enabled. True, this is unrelated to servers, which run without GUI, but still weird that it happens.
Also, when my laptop runs for weeks without reboot, it has issues with VRAM being occupied and not freed for some reason, which causes game crashes. With all the talk about Linux infinite uptime I expected better.
However, I almost can't complain about my homelab server. If there is enough disk space, it can run for months. Unfortunately, lately there are so many patches, I need to reboot almost each day (at least podman-hosting VM, if not proxmox host)
Re:Microsoft blames monster email outage on software defect ...
by vessenes on Monday November 16, 2026 @10:05AM
(#50518369)
Pretty interesting that testing was aimed the worry: low spec boxes, and missed a different problem: concurrency contention for high spec boxes. Gotta be covered by someone's rule of testing, I imagine. Also, woof.
Re:Microsoft blames monster email outage on software defect ...
by toss1 on Monday November 16, 2026 @10:09AM
(#50518404)
>> "the processes that led to an update with a latent defect reaching production deserve a long, hard look. "
Indeed they do deserve a long hard look.
Toss1's conjecture:
We are now seeing the beginning of an era of latent technical debt brought about by too rapidly firing developers and switching to AI code generation (nevermind vibe-coding). Even well-prompted and well-monitored by top expert coders and architects, there is latent technical debt, and it is more diffuse, harder to find, and harder to fix, and will start showing up in outages like this.
(It first seemed the ATC outage was another example, but it now seems it was merely the usual management incompetence if not an outright manipulative ploy to 'justify' more spending for a single-bid contract [0])
Re:Microsoft blames monster email outage on software defect ...
by nialse on Monday November 16, 2026 @02:31PM
(#50521907)
Reformulated conjecture: "Complex systems are kept stable by organisational knowledge of their weaknesses and how to compensate for them. Organisational change erodes this knowledge while the underlying weaknesses remain. A previously stable system thus becomes increasingly vulnerable to failure."
Re:Microsoft blames monster email outage on software defect ...
by bayindirh on Monday November 16, 2026 @10:36AM
(#50518712)
Also, don't forget that e-mail is a distributed system and servers retrying always keep a pressure on you, despite their best intentions, and spaced retries.
So, even when you fix the issue, it takes a bit to clear up, and the load gauge stays at the red zone for a while.
Re:Microsoft blames monster email outage on software defect ...
by lokar on Monday November 16, 2026 @10:45AM
(#50518835)
Anytime you run a proxy system for a large number of clients you don’t control you need a way to cold start. This means being able to allow traffic through (cheaply!) on a client basis, rejecting the rest.
This lets you start at a very small amount of load and gradually ramp up. But you can’t reject randomly or goodput stays at zero and the backlog never decreases.
Re:Microsoft blames monster email outage on software defect ...
by bayindirh on Monday November 16, 2026 @10:46AM
(#50518859)
Sounds reasonable.
I believe SDN systems can do that with ease, but what would be the equivalent method on older systems? Selectively allowing IPs to go through the trickle load into the system?
Re:Microsoft blames monster email outage on software defect ...
by toast0 on Monday November 16, 2026 @07:11PM
(#50525515)
Selectively allowing IPs is a reasonable way to do it. If you think about it, it's terrible, but it will get you going, and it's pretty easy do something like start with allow nothing, then allow all ips under 2.0.0.0, then 3.0.0.0, etc.
Distribution of users by /8 isn't very even and the order that customers will get online wouldn't be an order anyone would pick, but it should be reasonable to implement and synchronize (if needed) and it lets you make progress.
Re:Microsoft blames monster email outage on software defect ...
by DenisM on Tuesday November 17, 2026 @01:04PM
(#50535518)
I’m wondering if you’re preparing for cold start ahead of time you could create a better cold start map that’s distributed more evenly for more predictable turn up, and also with better affinity so you can you report progress in a way people can understand ( eg state by state). It would be a pain to load and activate the map into edge routers in a hurry, but maybe be necessary. Some superscalers are deep into writing their own networking firmware, this could go into design requirements for text revision.
Another idea is to have different clients use different ports (enterprise customers on one port, SMB on another) and turn up in the order of SLA.
Re:Microsoft blames monster email outage on software defect ...
by VirusNewbie on Monday November 16, 2026 @11:20PM
(#50527396)
sure, it's not that longer outages can't happen, but it's very peculiar that the outage was global, and that mitigation took that long. I would have expected fast mitigation for the most populous zones or regions.
The thing with plain outages vs. say, partial degradation of services, you can roll out fast because you're already at worst case scenario *[1].
Here's an interesting one to compare to that I'm very familiar with:
Re:Microsoft blames monster email outage on software defect ...
by tremon on Tuesday November 17, 2026 @12:27PM
(#50534928)
What do you mean, "not acceptable"? Given the situation, I think 15 hours is defensible. Where do you suggest putting the blame for the implementation and deployment of the fix?
What's not acceptable to me is a bug of this impact going undetected for 6+ months in production.
Re:Microsoft blames monster email outage on software defect ...
by nijave on Monday November 16, 2026 @08:20PM
(#50526164)
The irony is they had a more fault tolerant system 30 years ago with on prem Exchange. I'm not sure why they decided to make it a global mega service. Some things are a bit more justifiable with shared state requirements but email is stupidly distributed and fault tolerant.
You can't even say "but managing fleet/cells is harder": they make software for that, too!
Re:Microsoft blames monster email outage on software defect ...
by spprashant on Monday November 16, 2026 @10:20AM
(#50518525)
These kinds of incidents should inject healthy dose of techno-skepticism in the larger conversation. The world is teetering on the edge of turning over all our processes to LLMs and retiring human labor for good.
But at the end of the day these are servers strung together sending packets over copper wires. They are extremely reliant till they are not.
When crisis strikes you can only hope there is a squad of well trained humans who can step in to untangle these messes. But as more of the labor force is stripped out of responsibilities, the longer these outages will take to resolve. Imagine being jolted out of your year-long UBI-funded vacation to login and resolve a multi-continent server outage that you only have a vague memory of.
Re:Microsoft blames monster email outage on software defect ...
by greggoB on Monday November 16, 2026 @04:51PM
(#50523842)
From what I've observed, SV is doing anything and everything it can to make it reality. It's scary to see how successful their campaign has been so far.
Re:Microsoft blames monster email outage on software defect ...
by loboj on Monday November 16, 2026 @08:07PM
(#50526054)
Say I’m an IT manager and I choose a small provider for hosted Exchange. When they inevitably have an outage and we are the only ones down, I’ll hear from everyone up the chain about how I made the wrong decision, this is my fault, and clearly if we’d gone with the industry standard solution this wouldn’t be happening.
If I go with Microsoft and they have an outage, everyone is down and we can all commiserate about how it’s Microsoft’s fault and outside our control.
Re:Microsoft blames monster email outage on software defect ...
by ketzu on Tuesday November 17, 2026 @04:24AM
(#50529253)
Could you elaborate? The two obvious interpretations I see are:
* companies should have multiple parallel email and login systems (for everyone so they can switch when one is down)
* not all companies should use Microsoft hosted login and email services
But both aren't very convincing. Having parallel login systems seems like an awful idea and parallel email systems like a very expensive project with little upside (switching email systems on dns seems fun).
And most companies are already not on Microsoft, just a lot are. (it is also not obvious how that would improve the situation for the companies)
Re:Microsoft blames monster email outage on software defect ...
by p4ul on Monday November 16, 2026 @06:35PM
(#50525168)
Indeed! A "normal accident" as they say.
And it's also possible that these normal accidents might become more frequent with all the layoffs, AI code, and AI-generated exploits. I guess we'll see.
Re:Microsoft blames monster email outage on software defect ...
by ChrisArchitect on Monday November 16, 2026 @10:40AM
(#50518776)
This MS rando on X just leaking his 'abridged' version instead of letting the official account post the report/blog later or whatever? Gotta be 'first' right bro, and now the source of every rabid tech blog for the rest of the day. bah
Re:Microsoft blames monster email outage on software defect ...
by mike_hearn on Monday November 16, 2026 @10:42AM
(#50518809)
No big surprises here. Big outages at big firms seem to always look like this.
The electricity industry has this notion they call a black start, where they bring the grid online from zero. It's a very difficult operation because you can't start a generator straight into an offline grid otherwise it gets hit with full demand and can't keep frequency in the right range, which can damage equipment. So they have to start by disconnecting all the loads and then bringing load blocks back in sequence, trying to roughly balance available generation with the size of the next block. In practice black starts are extremely rare, but they can happen after a cascading failure in which something causes some generating stations to trip, redirecting load onto the remaining stations which can't handle it causing them to trip in turn. The entire grid can collapse in just 60 seconds or so once that kind of runaway failure starts.
This kind of outage is very analogous to a black start. A few servers get into trouble - maybe they're in a crash loop or maybe their caches are cold - the load balancers shift traffic onto the remainder which then can't handle the load either and so the failure cascades. Once a service is in this state sometimes the only remedy is to disconnect all load, bring the servers back to stability, then bring load blocks back online in small chunks, making sure to never hit anything with the full load immediately to ensure any caches or JIT compilers have time to warm up.
Black starting a service is obviously hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster. I used to be an SRE at Google and although we regularly started up new clusters and moved traffic around between them, actually starting an isolated service instance entirely from scratch was never something we practiced. I remember in a few cases teams discovered devs had introduced circular dependencies by accident, i.e. starting service X required service X to already exist, or it depended on service Y that in turn depended on X. Sometimes you can only find these things by practicing.
Re:Microsoft blames monster email outage on software defect ...
by ryanburk on Monday November 16, 2026 @08:30PM
(#50526236)
I can't speak for current MS cloud services, but we absolutely used to do this in the hotmail/outlook days. we had the notion of a "rack" of services that combined to compose the service plus a number of accounts, and before an update we would remove the scaling so it became more heavily loaded with real traffic patterns and see if it fit the current expectations or anything changed.
SBB just finished their project to be able to restart the grid in 5min. (There was a famous multi hour outage a few decade ago that I think motivated this work)
Re:Microsoft blames monster email outage on software defect ...
by mike_hearn on Tuesday November 17, 2026 @04:40AM
(#50529344)
I think 24 hours is more likely for the UK unless the outage was caused by a major storm that damaged equipment, or sabotage. A few isolated areas might take a few days to come back.
If the outage involves damage to transformers then all bets are off. In those cases some areas of the country might become uninhabitable for months.
Re:Microsoft blames monster email outage on software defect ...
by dubcanada on Monday November 16, 2026 @04:29PM
(#50523555)
That's not a 100% how this works, a load balancer setup should take into account a "server being ready to accept traffic" that includes cache and JIT.
A power grid is different because it's not about the power immediately being used all at once, it's about stabilizing the frequency. Turning generators on takes a while to stabilize, each reactor may output inconsistently for a while, which is why you never start a generator and immediately ask for full output, you start it, let it warm up, then add load. Which does sound like a server, but you don't need every server speaking the same frequency.
If a grids frequency is off, or stutters it kills electronics. If a servers JIT cache is ready, it's slow. The extremes are completely different.
Now the same garden hose problem exists, going from 0 to everyone logging in at once is a lot of load. But that's why you come back online with more servers and scale back.
Re:Microsoft blames monster email outage on software defect ...
by Rygian on Monday November 16, 2026 @05:02PM
(#50524006)
Back in 2025, Spain did a very-dark-brown-almost-black start (with help from neighbours), starting with one hydropower plant and increasing from there.
Re:Microsoft blames monster email outage on software defect ...
by tumetab1 on Tuesday November 17, 2026 @03:50AM
(#50529038)
In Portugal it didn't go that well, as expected since real situations differ from tests.
The initial start up (of the cold start plants) took the double of time expected.
The "cold start" capacity is being increased due to some of the issues found.
Re:Microsoft blames monster email outage on software defect ...
by ncruces on Tuesday November 17, 2026 @04:22AM
(#50529239)
It actually was better than in Spain. Large portions of Spain were left without power for almost 24h, despite the help from France (initially) and Morocco (later).
Portugal was able to recover in 12h (still a lot) without any external assistance.
Re:Microsoft blames monster email outage on software defect ...
by Rygian on Tuesday November 17, 2026 @07:14AM
(#50530376)
> About 16:00 CEST on 28 April, Spanish electrical operator Red Eléctrica de España (REE) estimated it would take "between six and ten hours" to restore service
> By 07:00 CEST (05:00 UTC) on 29 April, electricity had been restored to 99% of energy demand, and full restoration was achieved by 11:00 CEST (09:00 UTC) that same day.
The wikipedia references seem valid for this precise timeline, which is more informative than your remark.
I'm not trying to win a pissing match, just trying to counter the very common miserabilist notion that Portugal did worse than Spain.
This specific problem was (1) caused in Spain, (2) fixed faster in Portugal, and (3) without external help.
Portugal could've been the cause, and could've done better, for sure.
But it's just not balanced to write "In Portugal it didn't go that well" as a response to "It went remarkably well [in Spain]." Makes it sound much worse than it actually was.
Re:Microsoft blames monster email outage on software defect ...
by mike_hearn on Tuesday November 17, 2026 @10:16AM
(#50532510)
Power stations need power to start up and operate, called house load. It's for things like pumps and excitation current. To black start a station requires the house load to come from somewhere, like a generator set. For a coal, gas or nuclear plant the house load is quite large. For a hydro plant it's small, you just need enough to open the wicket gates and energize the generator coils. Therefore, black start capability is cheaper to sustain at hydro stations than other kinds.
Once the hydro station is back online the power it generates can be used to provide house load to the other stations.
Re:Microsoft blames monster email outage on software defect ...
by nialv7 on Monday November 16, 2026 @05:10PM
(#50524109)
that's not the problem with grid black start at all? black start is difficult because many types of generators themselves need power to start. it's a circular dependency problem, not a load management one.
Re:Microsoft blames monster email outage on software defect ...
by avianlyric on Monday November 16, 2026 @05:23PM
(#50524277)
It’s both, but every grid has a sub-set of power stations with black-start capability (a crap ton of diesel generators), that exists to break the circular dependency problem and bootstrap the rest of the grid.
But load management while bootstrapping the rest of the grid is still very hard. Often the reason for really long blackout, ones measured in days rather than hours, is repeated cascading failures as operators try to reconnect parts of a the grid, and accidentally overload recovered parts of the grid, causing generators to trip off again.
Re:Microsoft blames monster email outage on software defect ...
by tacticus on Monday November 16, 2026 @07:40PM
(#50525816)
> It’s both, but every grid has a sub-set of power stations with black-start capability
They should have that and they typically pay substantially more for it. Unfortunately it's also the case that the private companies providing said black-start capability may have just not bothered to tell anyone they haven't been able to do it for a few months\years.
Re:Microsoft blames monster email outage on software defect ...
by avianlyric on Wednesday November 18, 2026 @07:48AM
(#50545682)
Nor should. They absolutely must, otherwise it would be basically impossible to blackstart the grid.
A properly run network will require those facilities to be tested multiple times a year to ensure it actually works, and make payment for the facility contingent on passing those tests.
Re:Microsoft blames monster email outage on software defect ...
by cowlevel on Tuesday November 17, 2026 @04:20AM
(#50529223)
Wikipedia says black start stations are often hydro dams, since all they have to do is open the floodgates (literally), without any preheating or fuel pumps or anything like that.
Re:Microsoft blames monster email outage on software defect ...
by bell-cot on Tuesday November 17, 2026 @10:09AM
(#50532427)
Hydro is ideal - if you have a good-size hydro plant handy, where a sustained drought can't possibly drop the reservoir too low for the penstocks to function properly.
Re:Microsoft blames monster email outage on software defect ...
by avianlyric on Wednesday November 18, 2026 @07:47AM
(#50545665)
Can’t be any old hydro plant however. Big hydro plants will be using turbines is electrically excited magnets (I.e. electromagnets), but that’s only way to build very high power turbines without tons of expensive and fragile permanent magnets.
So you need a hydro plant that is either small enough that it’s using turbines with at least some permanent magnets (you get turbine varieties that basically have a mini turbine inside using magnets to power the energise the larger electromagnets), or has dedicated infra to produce power to energise the coils in their turbines before they’re producing enough power to energise themselves.
Re:Microsoft blames monster email outage on software defect ...
by bragh on Monday November 16, 2026 @05:34PM
(#50524431)
That was a very enlightening moment in Factorio when I had already converted everything to power and the whole grid blacked out and I needed to get it going somehow again.
Re:Microsoft blames monster email outage on software defect ...
by scheme271 on Tuesday November 17, 2026 @12:55AM
(#50527935)
The other issue is that you need to be able to sync up the generators. E.g. your power plants are running at 50Hz or 60Hz or whatever. But are they running in phase, if not, then how do you make sure that each new generator is in phase with the overall grid in order to avoid the generators from fighting each other?
Re:Microsoft blames monster email outage on software defect ...
by cowlevel on Tuesday November 17, 2026 @04:21AM
(#50529232)
For that they use a couple of analog knobs to adjust the generator speed, and a specialized type of analog meter called a synchroscope, which measures phase difference. When it's at zero and not moving, they pull the big lever.
This is very much like trying to jump onto a train from another train moving alongside it.
Re:Microsoft blames monster email outage on software defect ...
by mike_hearn on Tuesday November 17, 2026 @04:52AM
(#50529421)
Maybe in the 1920s ;) Pretty much any power station built post-war will have automatic synchronization. You aren't going to manually synchronize a large power plant these days, it's far too dangerous. Even very old plants have safeties that prevent closing the breakers if the sync isn't close enough.
Re:Microsoft blames monster email outage on software defect ...
by CobaltFire on Monday November 16, 2026 @05:10PM
(#50524116)
I never worked in civilian power generation, but I can vouch for this on Nuclear Ships.
We had to practice load pickup on a normal start at the pier often, but we likely practiced a full SCRAM to back online (including a FULL load shutdown) more.
Having predefined lists of what gets brought up at various points helps a ton, but it's still not an easy or simple operation.
Re:Microsoft blames monster email outage on software defect ...
by TomatoCo on Monday November 16, 2026 @06:29PM
(#50525088)
I imagine the predefined list is easier when it's all consumers that are under your control. I figure with civilian power some random percent of houses'll kick their AC on after power is restored.
Re:Microsoft blames monster email outage on software defect ...
by noir_lord on Monday November 16, 2026 @09:13PM
(#50526536)
It's why they bring up a subset a bit at a time and fewer houses/businesses than would be expected under average load, Practical Engineering has a fantastic video on black starts[1]
Re:Microsoft blames monster email outage on software defect ...
by lifeformed on Monday November 16, 2026 @09:58PM
(#50526837)
I never worked on nuclear ships, but I can vouch for this in Satisfactory.
You need a temporary source of power to start the water pumps and it takes a while for the water to flow all the way through the system and for the coal to saturate all the belts and power plants, if you hook it up to the main grid before all the plants are operational then it shuts down again.
Re:Microsoft blames monster email outage on software defect ...
by epiccoleman on Monday November 16, 2026 @10:35PM
(#50527095)
heh. this is why I've got programmable speakers hooked up to the belt that feeds fuel to the steam generators in my Factorio world - because I had one too many times where I moved something or messed up my logistics and had to cold start the base by hand feeding coal to get back online.
Factorio makes this fairly easy for the most part, a cold start isn't really some massive engineering challenge, but still, worth guarding against. By the time you get solar and / or nuclear power is basically solved, at least on Nauvis, but it's "overengineering the game," so... we overengineer.
Re:Microsoft blames monster email outage on software defect ...
by pjc50 on Tuesday November 17, 2026 @04:53AM
(#50529426)
Yeah, Gleba seems deliberately set up to make brownouts risky (letting your supply of trapped pentapod eggs hatch and eat your base) and black starts harder - base fills up with spoilage which needs to be removed. At least you can burn the waste for fuel.
In Factorio, solar is an easy source of cold start. In the real world, I think hydropower is a commonly designated choice and the solar farms have to wait for a grid to sync against.
Re:Microsoft blames monster email outage on software defect ...
by marcosdumay on Tuesday November 17, 2026 @12:10PM
(#50534587)
Power switches are your friend.
Cold starts are a constant in Aquilo until you research and build fusion power. And whatever you do by hand to handle them, there's a device in the game that does it automatically too.
Re:Microsoft blames monster email outage on software defect ...
by Beretta_Vexee on Tuesday November 17, 2026 @04:50AM
(#50529400)
It’s much the same principle as with an electricity grid, but on a different scale. We start up diesel generators to start gas turbines, which in turn supply power back to the distribution substations.
Once we’ve restored power to the substations, we can carry out what are known as voltage restoration operations.
At a nuclear power station, if all goes well, there will be at least one nuclear reactor unit that has managed to operate in island mode. They have their own sub station and can restart the others units.
We restart as many units as possible on a virtually empty section of the grid (serving only essential and vital consumers such as hospitals). We then reconnect consumers in a predefined order to avoid significant fluctuations that could trigger further cascading failures.
The most difficult part is often identifying the exact cause of the blackout with certainty, so as to prevent it from repeating it.
Re:Microsoft blames monster email outage on software defect ...
by nilptr on Monday November 16, 2026 @05:37PM
(#50524461)
There's literally no such thing as "black start" at Microsoft that I am aware of. The way services are brought online initially is through a build out process coordinated through hundreds of Teams. It's an organic, messy, and painful way of doing things. As such, build outs are a human driven process with bits of automation spanning sometimes 20+ years of engineering touches, troubleshooting guide, and some additional tribal knowledge thrown in passed down through the years. There is little consistency across teams and sometimes even within a team as every Azure Cloud is effectively slightly "different".
Re:Microsoft blames monster email outage on software defect ...
by nilptr on Monday November 16, 2026 @11:18PM
(#50527379)
Ha, no. It's not quite like that, but it can take weeks to bring up a service especially if you're in the lower rings. Often time core cloud stuff is being built out while services are also building out so stuff gets stubbed out and then you have to go back and slowly fill in the dependencies.
It's chaotic and stressful in a slow asynchronous way.
Re:Microsoft blames monster email outage on software defect ...
by coliveira on Monday November 16, 2026 @11:50PM
(#50527600)
Nobody should ever depend on things done by Microsoft. It's in their culture to create things this way, it is totally undependable. My guess is that we still haven't seen the complete mess that MS can create with their online services, and everyone will pay for this reliance.
Re:Microsoft blames monster email outage on software defect ...
by gosub100 on Tuesday November 17, 2026 @07:12AM
(#50530362)
It's so bizarre they have no consideration for UX, or reliability or quality. But they will keep 16-bit APIs for backwards compatibility to DOS because some peanut butter factory in Indiana still uses it.
Re:Microsoft blames monster email outage on software defect ...
by anankaie on Tuesday November 17, 2026 @01:59PM
(#50536440)
There is a BCDR strategy for each service at different tiers (based on which "ring [of criticality]" your service is in). Every service is supposed to pass through a (yearly? biyearly? don't remember) compliance certification. Not sure which rings require practicing it, though.
All of it is documented through a unified catalog of services and their associated metadata.
Source: Worked in MSR and shipped services through Azure / CoreAI.
Re:Microsoft blames monster email outage on software defect ...
by scheme271 on Monday November 16, 2026 @05:45PM
(#50524552)
Texas was on the verge of having to do a black start a few years ago when winter weather started causing failures. I think people were quoting potentially weeks to get everything back online. As it was, I think people didn't have power for days during sub-freezing temps due to load shedding.
Re:Microsoft blames monster email outage on software defect ...
by tcskeptic on Monday November 16, 2026 @09:55PM
(#50526807)
Yep. I was without power for 54 hours. Weirdly, just across the street from me never lost power because they were on a priority segment with a hospital.
Re:Microsoft blames monster email outage on software defect ...
by stephenhuey on Tuesday November 17, 2026 @12:44AM
(#50527876)
After winter storm Uri, my house was without power for 4-5 days (we had gone to a friend’s house which was in a small oasis of light in a dark neighborhood because it was on a high priority bit). Long before, I was in a house a little over a mile from downtown Houston during Hurricane Ike, and spent 7 days in darkness and heat and humidity while gazing at the bright skyscrapers a half hour walk from my pitch black street.
Re:Microsoft blames monster email outage on software defect ...
by toast0 on Monday November 16, 2026 @06:58PM
(#50525389)
> Black starting a service is obviously quite hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster.
I mean... you can try, but unless you spend a lot of resources on it, I don't expect test load to be close enough to really practice a black start.
From experience, the best way to prepare for restoration from nasty outages is to have them frequently! The team I was on was pretty good at outage restoration, until we started regularly hitting about a year between outages. After a few years of one or maybe two outages per year, we had lost the muscle memory for protecting services that get overloaded during or after an outage and setting outage messages (which both informs customers and protects customer service queues).
Regularly starting up an isolated cluster at least lets you get ahead of sequencing issues though; hopefully it's enough if you can get the whole cluster started and then modulate inbound traffic (most complex services will need something like you mentioned so that you limit the number of sessions, but sessions that start are likely to proceed successfully).
Re:Microsoft blames monster email outage on software defect ...
by hwillis on Monday November 16, 2026 @07:53PM
(#50525935)
> and can't keep frequency in the right range, which can damage equipment.
This is one of those beautiful engineer understatements. Rotational speed is fundamentally linked to voltage by the motor velocity constant: https://en.wikipedia.org/wiki/Motor_constants
Although it's fundamental for all motors, the actual constant is arbitrary for each motor. Voltage is still proportional to change in field is proportional to speed; at zero voltage you have zero speed.
A large dead AC load has its own speed- zero. It's connecting the engine without a clutch, but the engine makes hundreds of megawatts. The 6' wide generator wants to go from 3600 RPM to a standstill in a few inches. A full speed rotor can have up to 5 gigajoules of rotational inertia; over a ton of TNT.
Which is great, because connecting it backwards would cause a slow motion explosion. On a dead grid the only thing saving you is the resistance in the wires; it would be LOUD if you connected to it and you would very likely increase the ventilation through the walls of the plant.
Re:Microsoft blames monster email outage on software defect ...
by cowlevel on Tuesday November 17, 2026 @04:13AM
(#50529178)
That's for a different type of motor. Synchronous motors always run at line frequency no matter the voltage; induction motors always run at a bit less than line frequency. And stepper motors run at the step rate, of course they're not directly connected to the grid because that wouldn't work, but they are also voltage-independent motors.
Re:Microsoft blames monster email outage on software defect ...
by arghwhat on Tuesday November 17, 2026 @12:40PM
(#50535118)
nit: all motors follow their fields, and no motor is implicitly tied to mains grid frequency.
That we often bypass local field control and tie large industrial synchronous and asynchronous (induction) motors directly to grid is narrow and quirky special case that is common for historic raisins but really quite awful, with staged delta-star or dahlander winding relay tricks to go from insane start currents down to just absurd.
Brushed motors are just synchronous motors with a primitive mechanical field control attached, allowing them to reduce the field rate in response to load.
Re:Microsoft blames monster email outage on software defect ...
by trollbridge on Tuesday November 17, 2026 @08:45AM
(#50531300)
Note that going to solar and power storage fixes a lot of this since it has far more control over synthesising the frequencies needed, as opposed to big spinning turbines with lots of inertia.
Re:Microsoft blames monster email outage on software defect ...
by bdamm on Tuesday November 17, 2026 @12:41PM
(#50535139)
Ah, no. Solar and wind are great energy sources, but electronically synthesized sine waves (what you get out of a solar inverter) can also have feedback harmonics that can build into destructive frequency waves. It's not a huge problem as long as the majority of generation is backed by rotating mass, but increasingly power islands and ever more distributed solar generation is tipping the scale of that equation, so we're likely to see more of these sorts of distribution and transmission issues caused by poorly behaving frequency generators' emergent behavior.
Re:Microsoft blames monster email outage on software defect ...
by aforwardslash on Tuesday November 17, 2026 @12:54PM
(#50535352)
Actually, its the opposite; afaik, injecting sythesized ac requires an already running ac system so it can match the frequency. Its non-linear characteristics can cause catastrophic imbalances in the grid - see portugal/spain blackout.
Re:Microsoft blames monster email outage on software defect ...
by andix on Monday November 16, 2026 @08:01PM
(#50526000)
External load can be blocked/limited extremely easily for an internet service. Just block all incoming connections or limit ip ranges at firewall level. Load can be incrementally increased by unblocking more and more ip ranges.
(It might be better not to use CIDR ranges, but use the lower bits instead if possible. For example starting to unblock all IPs that end with .80 until .90, to get more random geographical distribution)
Which leads to my suspicion it was internal load that brought the system down. Communication between the servers (like delivering queued up messages from one O365 account to another), and not external access from users and apps.
Re:Microsoft blames monster email outage on software defect ...
by dboreham on Monday November 16, 2026 @08:57PM
(#50526426)
Quick note that sometimes it's not that simple. I've worked on a somewhat similar system (although overall much smaller in scope). What we found was a) clients hop between networks and hence IP addresses quite frequently: mobile devices. And b) with sync type protocols you can't just randomly serve whatever sliced and diced subset of the incoming traffic you like. What happens is that many devices end up never achieving current sync state, so they re-try. In our service we had to implement a protocol-aware, and device ID-aware filtering scheme. It had the property you mention where you could dial the proportion of load up or down, but it did it across the known population of devices: if a device got white-listed at t=0 it remained white listed for the remainder of the day. Once it achieved up to date sync it got put into another load bucket. The bucket of devices transitioning from out of sync to synced up was what we limited. It was only after we implemented such a scheme that we got past our thundering heard / can't bring the service back up after an outage problems.
Re:Microsoft blames monster email outage on software defect ...
by andix on Monday November 16, 2026 @09:29PM
(#50526655)
I don't get this issue. The vast majority of clients would not change IP during a short sync that takes maybe a few seconds. And a switchover from an unblocked to a blocked IP would either result in a partial or in a failed sync, without any consequences.
If the load balancer already does some connection tracking, it can just keep serving existing connections and only drop new ones. If connection tracking is not available, the IP blocklist would be a suitable option.
Re:Microsoft blames monster email outage on software defect ...
by andix on Tuesday November 17, 2026 @08:45AM
(#50531288)
> IPs on every sync on some networks because of carrier NAT
Okay, they get different IPs during the day, fine. But it's still kind of randomized. Stepping through IP ranges will reduce load and give access to more people.
I really don't get the issue you are seeing with this.
Re:Microsoft blames monster email outage on software defect ...
by psd1 on Tuesday November 17, 2026 @08:55AM
(#50531418)
IP blocking is dirt cheap on your edge routers. Application-aware binning needs at least your app gateway to have spare capacity. I can see that going tits-up.
Re:Microsoft blames monster email outage on software defect ...
by tremon on Tuesday November 17, 2026 @07:37AM
(#50530583)
> Which leads to my suspicion it was internal load that brought the system down
IIRC Microsoft posted a comment around 10pm where they discussed rate-limiting external connections, and many reactions on last week's HN thread were like "wait, you so didn't even drop load before now?" So internal load may have been a contributing factor, but it's by no means certain given that they didn't stop serving clients.
Re:Microsoft blames monster email outage on software defect ...
by drusepth on Monday November 16, 2026 @08:11PM
(#50526086)
> In practice black starts are extremely rare, but they can happen after a cascading failure in which something causes some generating stations to trip, redirecting load onto the remaining stations which can't handle it causing them to trip in turn. The entire grid can collapse in just 60 seconds or so once that kind of runaway failure starts.
This is absolutely terrifying knowledge to know given the competency of the current administration and their attitude/response to recent "black swan" outages.
Re:Microsoft blames monster email outage on software defect ...
by water42 on Tuesday November 17, 2026 @12:05AM
(#50527675)
why did you feel a need to explain cold starts, a standard distributed systems problem, with an elaborate story about "black starts" from an analogous space? you can just explain what cold starts are directly and talk about the relevant problems that come up when doing so.
Re:Microsoft blames monster email outage on software defect ...
by jdironman on Tuesday November 17, 2026 @12:12AM
(#50527704)
i found their comment useful as someone who works in tech but doesn't always understand things at scale like what they explained. So it increased my understanding and actually made me think about some of my own work from a different angle versus say reading a comment critiquing using analogies.
Re:Microsoft blames monster email outage on software defect ...
by water42 on Tuesday November 17, 2026 @03:04AM
(#50528726)
Readers would understand what happened better given a distributed systems example. Lots of problems come up cold starting distributed systems that aren't explainable through "black start"
Re:Microsoft blames monster email outage on software defect ...
by balarjin on Tuesday November 17, 2026 @11:12AM
(#50533516)
> Readers would understand what happened better given a distributed systems example.
Readers who know about distributed systems and their failure modes already understand the thing being explained. They (and you) are not the target audience. Let people who are not already experts learn something by analogy.
Re:Microsoft blames monster email outage on software defect ...
by mike_hearn on Tuesday November 17, 2026 @03:37AM
(#50528948)
Well, mostly because it just occurred to me and I thought it would be worth sharing.
But the analogy is useful in another way. The electricity industry plans and rehearses for this kind of thing, even though it's impossible to simulate. I used to be rather skeptical that these rehearsals would be effective but Portugal black started last year after a solar plant in Spain malfunctioned and took down the whole Iberian grid. It took almost exactly 24 hours despite the grid collapsing several times during the process, which is the usual estimate for how long things will take. So it seems they have this stuff well under control.
I won't slam Microsoft too hard for this outage because this type of scenario has played out at many companies over the years. Yet I never encountered an ops team that has written procedures for how to recover when the entire service has entered a crash loop under peak load.
The exact analogy we use for this doesn't matter much as they are all pretty evocative, but using black start has the advantage of reminding us that there's another branch of engineering where this necessity is taken seriously and rehearsed. They also engineer for it ahead of time so they can prioritize customers like hospitals and airports under overload conditions. I think it would be a good feature for cloud LBs to be routinely differentiating between e.g. customers on paid plans vs free so they can serve paid customers first if there aren't enough live backends.
Re:Microsoft blames monster email outage on software defect ...
by water42 on Tuesday November 17, 2026 @04:01AM
(#50529125)
>Yet I never encountered an ops team that has written procedures for how to recover when the entire service has entered a crash loop under peak load.
this is an interesting observation, since this should be table stakes for running a high scale distributed system. there are plenty of companies that do this and write about this extensively. it definitely becomes less important until you get to certain levels of scale though - a random app with 10k users has a different standard than, e.g., google cloud storage.
Re:Microsoft blames monster email outage on software defect ...
by mike_hearn on Tuesday November 17, 2026 @04:44AM
(#50529372)
Like in any such failure it's an onion you can unwrap several layers deep. The initial cause was a solar plant where the inverters malfunctioned and started dumping unsynchronized power into the grid in an unobservable way (grid operators didn't know where it was coming from). Their attempts to fix the issue caused other problems which combined with some plants being too sensitive and tripping when they perhaps didn't need to.
At the time Iberia was relying >50% on renewables which is very dangerous. The operators tried to recall gas power plants to provide inertial stability, but the plants were cold as they'd been taken offline in the morning to make way for solar (prioritized due to green politics), and even though gas plants can spin up very fast compared to other kinds they could not warm up fast enough to rescue the situation as it was all over thirty seconds later.
So it depends how you define "fault" but yes the starting point was a broken solar plant. Thermal/hydro plants stay locked together through physics because there's a prime mover connected to a generator, but solar plants rely on computer-managed simulations of the rotation that tries to follow the rest of the grid. When you have more simulation than reality it can get tricky fast.
Re:Microsoft blames monster email outage on software defect ...
by mike_hearn on Wednesday November 18, 2026 @04:06AM
(#50543808)
No, that's just a neutral writeup of what happened.
Less neutrally, there have been far too many years of people pretending that you can build a grid out of 100% renewables. It was a false claim from the beginning, but engineering was overridden by ideology. The Iberian outage was one of the consequences - unfortunately merely a preview of coming attractions, as our political class is in denial and would rather believe lies than address the truth that it's caused by their own laws. Now it's only a matter of time until the entire UK or European synchronized grids experience a collapse of their own, with a similar root cause. We can't say we weren't warned.
Re:Microsoft blames monster email outage on software defect ...
by Epa095 on Tuesday November 17, 2026 @01:27AM
(#50528105)
Its only analogous in the sense that we must have capacity ~>= load, and that you can get cascading failure. Where the analogy breaks down is that in power production you also have that capacity ~<= load, so you must have load and capacity roughly balanced. But that's not the case for the outlook servers, nothing explodes if there is 1000 over-provisioned servers.
Re:Microsoft blames monster email outage on software defect ...
by pyrale on Tuesday November 17, 2026 @03:12AM
(#50528774)
If we are to add to the power grid analogy, black starts are so rare because there is a lot of work going into mitigating failures.
For instance, a generator failure that cannot be mitigated by dropping load will probably result in islanding the grid: circuit breakers trip some lines and your network becomes several sub-networks. The sub-network that tripped can then be re-powered without a black start.
Re:Microsoft blames monster email outage on software defect ...
by wvh on Tuesday November 17, 2026 @05:27AM
(#50529661)
In my experience, we too often try to engineer around unavailability with caches, queues and cold serving, instead of just telling clients they can't have the cookie and have to wait. Maybe LBs should have a random drop policy in case of extreme overload of backend services, enough to keep things working somewhat both internally and visibly for clients, instead of contributing to the stampede and the whole system subsequently going completely down for all. "Too many" is a very real state.
An important part is that clients pre-fail most requests and don't send them if the server is overloaded. They just assume the server would return an overload error.
Re:Microsoft blames monster email outage on software defect ...
by trollbridge on Tuesday November 17, 2026 @08:48AM
(#50531334)
This has been understated; at Google everything had to be engineered for a cold start. And they’d actual go test this periodically, using techniques to do so so end users never knew.
1. They did not have proper cell-based architecture to partition their system into failure domains ("In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure.").
2. They did not have proper monitoring to see the encroaching collapse.
3. Stuff just keeps falling off: "Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day"
Re:Microsoft blames monster email outage on software defect ...
by k0morebi on Monday November 16, 2026 @04:33PM
(#50523610)
Not a network engineer so genuinely curious how these things are handled at the epic scale of 365, azure, aws and so on.
Extent of my knowledge is nginx or traefik load balancing which at this scale is virtually equal to no knowledge. But seems like there would be systems in place to simply drop a sliding percentage of excessive traffic or block some regions once critical thresholds are reached rather than compounding the problem with forwarding the growing number of requests downstream to next victim server.
Re:Microsoft blames monster email outage on software defect ...
by materielle on Monday November 16, 2026 @11:41PM
(#50527526)
There’s all sorts of way to do it.
The easiest, and the default choice when I worked at Google, was exactly what you said: a sliding scale.
You could target racks, data centers, clusters, regions, etc. And “drain” the unit. For instance, stop routing traffic to cluster foo.
Then you could increment up stepwise 10%, 20%, etc.
Obviously we didn’t use nginx, and sure there’s interesting engineering involved. But it’s not like these systems are magic. There’s just a lot of clusters and a lot of load balancers. But at the end of the day, it’s just a load balancer.
As another Xoogler in this thread pointed out, there’s two problems:
1) Sometime teams accidentally introduce circular dependencies. Service foo might require service bar to boot and vice versa. Typically that’s fine, but from a rare cold start where both foo and bar are down, you’re in a pickle.
2) Load balancing is “easy”. But the services need to be designed with the aforementioned load balancing strategy in mind. Stuff gets tricky if there’s any pollers, process affinity, multiple components that need to process parts of the same user’s request in tandem. Given that outlook is ancient, I would not be shocked to learn that the system doesn’t respond well to naive “slowest ramp up traffic to a cluster” approaches. For big distributed applications, it takes a lot of meticulous design to build a system that responds well to simple SRE techniques like that.
That is one thing my team at Google would actually consider. We would make the code and architecture more complicated to accommodate the types of naive mitigations one would want to apply in an outage.
Re:Microsoft blames monster email outage on software defect ...
by mindwok on Monday November 16, 2026 @04:49PM
(#50523819)
The biggest surprise here is that they're using Windows for critical network infrastructure. I'm surprised a Windows process has any role here outside the Exchange servers themselves.
Re:Microsoft blames monster email outage on software defect ...
by devy on Monday November 16, 2026 @05:21PM
(#50524249)
Not a surprise. It's Microsoft, Windows is their bread & butter for a long time. And Windows Kernel team is actually full of amazing engineers, reputable computer scientists. Disclaimer, my former boss worked at Windows Kernel team, she's the most talented engineers I have ever worked with, she then went on lead core platform engineer at Shopify many years later.
Before Linux took over the server market in 2000, Windows server was the dominant player in mid-market (enterprise market has IBM mainframes too). But you would have be old enough to remember that decades ago.
Re:Microsoft blames monster email outage on software defect ...
by 72deluxe on Tuesday November 17, 2026 @05:42AM
(#50529754)
I used it from early on but couldn't work out why it was so popular. Some of the extensions were really very useful but each release seems to shoehorn more AI things in.
Re:Microsoft blames monster email outage on software defect ...
by spartanatreyu on Monday November 16, 2026 @06:49PM
(#50525295)
C# was great, Typescript was great (especially considering it also needed to support JS' historical baggage), VSCode was great.
Typescript paused new feature development to port to a new implementation language. It's too early to see if it's going to be better or worse for TS devs, will probably need to wait a few years (at least after the easiest things to pick are gone) to see if it was worth it for anyone who isn't building vscode sized projects.
VSCode was great, until LLMs hit the scene. Now there's a new update every 2/3 days either shoving more LLM-ware to the point that the seams are breaking, or it's a security bugfix for something that should never have been added in the first place.
Re:Microsoft blames monster email outage on software defect ...
by anonymous908213 on Monday November 16, 2026 @07:01PM
(#50525408)
I use a pre-LLM VsCodium fork. Works fine, I think VsCode was already feature complete years ago and you can easily extend it yourself if you need something it doesn't provide out of the box.
Re:Microsoft blames monster email outage on software defect ...
by netcoyote on Monday November 16, 2026 @08:50PM
(#50526376)
SQL Server is amazing technology. Spent months fiddling with IBM DB2 to get it to perform like out-of-the-box SQL Server. Didn’t like the pricing much, but the software was excellent.
Re:Microsoft blames monster email outage on software defect ...
by mindwok on Monday November 16, 2026 @06:11PM
(#50524908)
I have heard this repeated a few times now and it surprises me. Since going deep on Linux I've resented any time I had to use Windows, but folks I really respect often speak highly of the kernel. It'd be cool if they open source it one day and we get a good deep dive into the internals.
Re:Microsoft blames monster email outage on software defect ...
by makeitdouble on Monday November 16, 2026 @06:31PM
(#50525113)
Another angle on this: most ATMs were running windows a decade or two ago (I'd assume most still do, but not sure).
You'd see occasional crash screens, but given the sheer number of machines out in the wild, having only a few crash once in a blue moon speaks amply to the reliability of the thing.
Re:Microsoft blames monster email outage on software defect ...
by pasttense01 on Monday November 16, 2026 @07:56PM
(#50525960)
There is a growing problem with something called ATM Jackpotting often with this old Windows installed software:
"The FDIC OIG is alerting banks about an increase in a type of fraud known as “ATM Jackpotting.” ATM Jackpotting involves criminal actors accessing ATMs and placing malware software on the system, enabling them to illegally dispense cash from the bank’s cash supply. The process is often run by sophisticated and organized crime rings at both the physical location (accessing machine and hard drive) and operating in foreign jurisdictions (transmitting malware codes).
The FBI estimates 700 instances of ATM Jackpotting and losses totaling more than $20 million in 2025."
Re:Microsoft blames monster email outage on software defect ...
by gerdesj on Monday November 16, 2026 @07:58PM
(#50525971)
An ATM has at most a number pad with a few extra keys and some keys at the left and right of the screen or a touchscreen with a few buttons. That is rather simple compared to a full multi user system with actual multiple users and multiple apps.
An ATM is actually simpler than a fruit machine. The latter has more reasons to refuse a payout!
Re:Microsoft blames monster email outage on software defect ...
by jamesfinlayson on Monday November 16, 2026 @08:31PM
(#50526245)
Yeah I remember seeing an ATM that was showing a Windows screen five or ten years ago. Based on having seen some screens on the ATM before it must have gone live in... 1993 at the latest.
Re:Microsoft blames monster email outage on software defect ...
by notstayinglong on Monday November 16, 2026 @07:31PM
(#50525740)
The NT kernel was designed in the 1990s, whereas the Linux kernel is more or less a straightforward continuation of a design from the 1970s. NT does genuinely incorporate useful innovations over Linux.
Unfortunately, as I understand it, it basically needed to pretend to be Win 9x on release due to Microsoft's (understandable and smart) focus on backwards compatibility, so there's effectively a layer on top of NT that hides its most powerful capabilities. Then there's all the cruft Windows layers on top of the kernel, so one could be forgiven for missing the gem sitting beneath the mound of manure.
From what I've read, Windows NT does have a good design underneath while Unix is simpler, but I don't think any windows version has offered the capabilities in a straightforward manner and easy to administrate. Too much awful layers as the parent comment says. While Unix has been mostly the kernel => the service manager => your service surrounded by a few concepts like users, unix sockets, file permissions,...
Re:Microsoft blames monster email outage on software defect ...
by torginus on Tuesday November 17, 2026 @06:02AM
(#50529875)
Windows has been traditionally easier to administer in a standardized way. From the very beginning, it had a concept of system event logs, ACLs, service scheduling built into the OS.
These technologies are battle-tested, and were adopted or replicated in Linux-land . Fore example, systemd does event logging and service management. Likewise, Linux file permissions haven't really stood the test of time, enterprise-grade systems use ACLs, which again, are a retrofit into Unix/Linux, but native to windows. For Linux, you have a choice of these implementations, but most are built on top of the OS, and each have their fanbases.
The 'traditional way' of sysadmin-style semi-custom Linux boxes, each with their own permissions setups and init scripts have thankfully gone out of vogue.
Re:Microsoft blames monster email outage on software defect ...
by adrian_b on Tuesday November 17, 2026 @10:00AM
(#50532281)
ACLs come from Multics, decades before Windows NT.
The creators of UNIX inherited the hierarchical file system of Multics, but they have abandoned ACLs, considering that they are too complicated for small mini-computers with a few users, and they have replaced the ACLs with the well known UNIX file permissions.
So ACLs are not some Windows or Microsoft innovation. Both the POSIX ACLs and the Windows NT ACLs have resurrected the Multics ACLs.
Regarding system logs and service scheduling, Windows has made only negligible improvements over UNIX.
Re:Microsoft blames monster email outage on software defect ...
by gerdesj on Monday November 16, 2026 @07:52PM
(#50525922)
"The NT kernel was designed in the 1990s, whereas the Linux kernel is more or less a straightforward continuation of a design from the 1970s. NT does genuinely incorporate useful innovations over Linux."
Neither are really recognizable from their "roots" these days. The only certainty is that updating Windows still takes bloody ages.
Today I took an Ubuntu VM from 24.04 to 26.04 - it took about 10 minutes (that includes downloading the updates). It was a fairly simple system and I didn't have to repair anything. The service was actually up for most of the update.
I've run Exchange updates that have lasted most of an evening and night and that was on a single box Exchange on prem deployment.
On Gentoo Linux an emerge -Uva --deep --newuse @world can take a little longer than a Windows update sess but you are compiling the OS from source 8)
Re:Microsoft blames monster email outage on software defect ...
by dagss on Tuesday November 17, 2026 @01:07AM
(#50528007)
This thread is about the kernel. In your update process the kernel is only one of those Ubuntu packages. I bet the Windows update is also mostly stuff that is not the kernel.
Re:Microsoft blames monster email outage on software defect ...
by axmebhe on Tuesday November 17, 2026 @03:17AM
(#50528808)
File operations are surprisingly slow on Windows, and the reason is in the kernel. Compare git operations on a large repo with Linux or Mac, for example.
Re:Microsoft blames monster email outage on software defect ...
by cowlevel on Tuesday November 17, 2026 @08:28AM
(#50531076)
Indeed, and it makes sense since Windows's history involves more large independent files rather than zillions of tiny ones. The man pages for example - on Linux it's a file per page, on Windows it would've been a single hlp or chm. Programs too, they tended to be more statically linked.
Re:Microsoft blames monster email outage on software defect ...
by hn_acc1 on Monday November 16, 2026 @07:59PM
(#50525980)
And NT was basically a straightforward continuation of VMS from the 1970s which was, surprise, surprise, written by Dave Cutler when he worked at DEC and then went on to be lead architect of Windows NT.
DEC even sued Microsoft and Microsoft settled (sound familiar? DOS? Can't Microsoft invent anything of their own?).
Hey, they got it quite reliable eventually (Windows 2000 timeframe), but still.. To say Linux is 1970s tech and WinNT ISN'T seems a real stretch.
Re:Microsoft blames monster email outage on software defect ...
by hobo123 on Tuesday November 17, 2026 @01:46AM
(#50528225)
It's also tired, as if no kernel ever innovated or improved things. Today's Windows or Linux probably evolved in some ways from the old ones.
And if other more modern kernels offer significant improvements over them, I wonder why we are not seeing them. Should be easy enough to add a Linux or Windows syscall layer to a modern kernel like that.
Re:Microsoft blames monster email outage on software defect ...
by cowlevel on Tuesday November 17, 2026 @08:26AM
(#50531058)
compatibility isn't about the syscalls, but all the stuff they do. Somewhere in your stack is probably an exec call to a POSIX utility with a flag that's only supported by GNU coreutils. When Debian switched /bin/sh from bash to dash (more POSIX compliant), it broke a lot of stuff.
Re:Microsoft blames monster email outage on software defect ...
by masfuerte on Monday November 16, 2026 @10:03PM
(#50526864)
Windows NT 3.1 came out two years before Windows 95. The API for the GUI stuff was a straight port of the 16-bit Windows 3.1 API (hence the name) but much of the basic OS stuff (like the API for accessing files) was new.
Re:Microsoft blames monster email outage on software defect ...
by usef- on Tuesday November 17, 2026 @02:02AM
(#50528337)
I believe the grandparent is thinking about Windows 2000, where they added all the compatibility to the NT kernel to match the 9x series (eg, dos direct hardware access).
Re:Microsoft blames monster email outage on software defect ...
by adrian_b on Tuesday November 17, 2026 @09:52AM
(#50532173)
The NT kernel was well designed, but it did not have many significant innovations.
In the beginning, the NT kernel API was a mixture a functions taken from VMS (mostly for asynchronous I/O and thread/process management) with functions taken from UNIX (mostly for classic I/O), where most functions had various corrections or enhancements in their lists of parameters in comparison with their models from VMS and UNIX.
The later Windows NT versions added some innovations, e.g. I/O completion ports in Windows NT 3.5, waitable timers in Windows 4.0, and so on, but there were not many such innovations and most were not very important.
Around 2000, one could claim that Windows NT is a much more modern OS than Linux.
However, after 2000 Linux started to add a lot of innovations over what is specified by POSIX. Today, any high-performance application running on Linux that uses the POSIX API instead of using Linux-specific features, makes a big mistake.
Nowadays, for many applications Linux has a better API than Windows, but the Microsoft developers are not stupid, so in recent years they began to incorporate into the Windows kernel some of the more important Linux kernel innovations, like futexes and io_uring.
Re:Microsoft blames monster email outage on software defect ...
by cowlevel on Tuesday November 17, 2026 @04:25AM
(#50529263)
NT has fully asynchronous I/O while Linux usually just blocks a thread for anything that isn't a network or a pipe.
Somehow it manages to be slower in uncontended cases so maybe in hindsight it wasn't that good. But on Linux you get permanently stuck threads if your NFS server is down.
Re:Microsoft blames monster email outage on software defect ...
by mike_hearn on Tuesday November 17, 2026 @05:21AM
(#50529616)
It's not that great. The idea it's amazing is a bit of a meme. There are many things about the Windows kernel that are much worse than Linux.
Consider: exclusive file locking for almost everything. UNIX lets you delete or move an open file, the Windows kernel doesn't and this creates enormous complexity at higher levels of the userspace stack especially for any kind of auto updater. It's also one reason Windows updates are so painful.
Windows has pathetically low performance in the filesystem layer, largely thanks to its filter driver design. It's not uncommon to find apps written originally for Linux are basically un-portable to Windows because they assume fast file IO and Windows just can't provide it. This was also a major motivator of the WSL2 effort. Windows actually has a feature called Dev Drive. It's a whole alternative filesystem that's faster than NTFS. They call it Dev Drive because they assume only devs need good filesystem performance (gah). You can't run Windows itself on a Dev Drive though, because of tech debt-y "reasons".
This ridiculous approach of assuming developers are special is carried over to other things too. Windows supports symlinks but this support is nearly useless, because it's disabled out of the box for security reasons (symlinks play a role in some kinds of attack on Linux). So you have to turn on Developer Mode in the control panel to be able to use them.
Sometimes you'll see the Windows approach to security be praised (SIDs, object manager etc). It does have a more principled design than Linux, that's true, but the standard here is macOS/XNU not Linux and the Windows kernel's sandboxing support is far worse than macOS.
Windows processes are extraordinarily heavy compared to a Linux process. They take more RAM, start up much slower, and context switching is slower. Again, this difference often renders Linux software un-portable because they assume they can fork and exit processes very fast, but on Windows you just can't. You pretty much have to replace a multi-process design with threads.
Despite the Windows kernel not being a competitive advantage to Microsoft (IMHO), and despite that they run public symbol servers, they keep it closed source. If you really want to understand the Windows kernel you can do so down to the level of individual function calls with public information, but you can't actually read them. The reason is probably security-through-obscurity for DRM, Warbird and other such things, not that it seems to help much. This makes it frustrating to figure out what's going on.
Relatedly, the Windows kernel is an absolute war zone. It's nothing like Linux, which is a conventional textbook OS design. On Windows the kernel is riddled with deliberate traps, code obfuscation engines and "tilt sensors" designed to try and block attempts to hook internal functions by malware or other kinds of malicious software. This shows up in other places too: for example, the API to Linux is the system call instruction. On Windows the kernel theoretically doesn't have a stable syscall interface and you're supposed to go via public APIs like kernel32.dll, which in turn just forwards to ntdll. In practice it does have stable syscalls because MS never randomized the system call interface, so apps trying to detect hooking in userspace do invoke system calls directly (often for anti-cheat purposes). The result is there are like three different ways to invoke NT system calls, all of which are extremely overlapping and duplicative (kernel32, ntdll, system calls), all of which are used by apps in reality but only one of which is documented.
Re:Microsoft blames monster email outage on software defect ...
by torginus on Tuesday November 17, 2026 @07:24AM
(#50530462)
I would contest your contestation. While I admit that Windows suffers from lack of engineering attention as of late compared to Linux, but many of these things
- File locking is a convention on Windows, not a necessity and you don't pay for it if you don't lock files (which is an unfortunate default btw).
- By filter drivers being slow, I think you mean antivirus and security stuff, which are an unfortunate reality of daily windows use, but aren't really caused by filter drivers being a thing. If you needed antivirus on Linux, I think you might incur a similar penalty
- I think symlinks would be nice, but I haven't ever really missed them on Windows. The things linux uses it for, like linking libraries are done differently on Windows
- Devs are special in some sense. You can't afford to break existing workflows on users machines. Just recently, the EU decided to replace Windows with Linux, discovered that stuff doesn't work, then promptly went back
- I appreciate that you call out Windows security as a positive, but as with many other subsystems, Windows has shipped with many systems that were production grade and built in, like security, ACLs, event logs, etc. etc. which are not part of Linux by default, and are the subject of endless debates as to how to implement them (some of them, like systemd, mirror Windows functionality)
- A concept of a Windows process doesn't really map to a Linux process. Linux processes are isolated threads, while Windows ones are containers (and don't even necessarily have a main thread). And afaik Windows processes aren't that heavy any more. Their heaviness came from loading gdi32 and user32 dll, as they expected to host GUI apps, on new Win versions, that's no longer necessary. But fork() style process management is foreign to Windows, and the one place where its useful in Linux, cli apps, Powershell has an entirely different design, where its more like a JITed scripting language, and ls or cp are not separate processes.
- I don't think heavily multiprocess designs are in vogue anyways (at least not how Linux does them), but in terms of RPC, COM multi-process RPC was much more efficient in Windows compared to things like eventfd. Android even added binder, which is a similar mechanism
Many of Windows' mechanisms are a bit long in the tooth, but arguing that Linux is better, then adopting a very similar mechanism in Linux years to decades later doesn't really show that engineering superiority.
I'm not a Linux hater btw, except in one thing - the fact that there's no good standard way to build and distribute binaries that can be expected to work on any reasonable Linux install is criminal.
Re:Microsoft blames monster email outage on software defect ...
by mike_hearn on Tuesday November 17, 2026 @10:27AM
(#50532680)
A few unimportant quibbles... you can't avoid locking files on Windows because the OS locks EXEs and DLLs automatically.
IIRC it's more that the filter architecture itself is slow, but yeah Windows comes with some slow filters out of the box that don't help. There's a writeup somewhere about this from a Windows kernel engineer but it was years since I read it. AV is indeed the worst offender but Windows always has one, so <shrug>.
systemd is inspired by macOS Launch Services. I guess svchost is the closest equivalent in Windows, but it didn't play any role in the systemd design process.
Linux processes do contain threads, I didn't quite understand what you meant by saying they're not containers. There is now the concept of a Windows picoprocess but it can't do much and isn't used after WSL1 was removed, if I understand correctly. I agree that Windows doesn't really need cheap processes architecturally and native apps don't rely on them without that imposing much cost.
[D]COM isn't a kernel feature but the closest equivalent on Linux would be DBUS, not eventfd. I don't know how they compare in terms of efficiency, I'd expect about the same, although in a past life I actually worked on an open source implementation of DCOM!
I didn't argue Linux is better, I have plenty of beefs with how Linux works too :) The Windows kernel is rightly very popular because it offers a stable driver API, which in turn enabled a huge gaming and hardware ecosystem, for instance. But architecturally I don't see big differences between them tbh.
Now Midori, on the other hand... that really was different!
Re:Microsoft blames monster email outage on software defect ...
by cowlevel on Tuesday November 17, 2026 @08:33AM
(#50531132)
A lot of this cuts both ways. Linux not locking files creates as many problems as it solves, as do hard links.
I see that in general Windows takes a "big monolith" approach to things that Linux can handle as a large array of small pieces. An app can be a few files on Windows, a help database is one file, a source code repository would be one file if all the VCSes weren't imports from Linux.
This might be Conway's law in action. On Windows you were more likely a proprietary shop writing software that just does what it should do - if you needed to change it then you changed the software. On Linux because it's a bazaar you have a lot more of trying to tinker with the things that are already there, or add on small pieces. The average unit of deployment on Microsoft platforms is (or was) Turbo Pascal or Visual C++ 6 or QuickBooks or Microsoft Office; the unit of deployment on open-source platforms is grep or sed.
Re:Microsoft blames monster email outage on software defect ...
by hn_acc1 on Monday November 16, 2026 @07:54PM
(#50525940)
By mid-market, do you mean "internal to a company file server / active directory / print server / Exchange server" stuff? Sure.. Public facing servers (web / email / databases)? Seems like a stretch TO ME to say Windows was ever the dominant player in any major server market - SunOS/Solaris was by far the biggest server platform for startups / dotcom boom from say 1995-2002 so if you're saying Linux took over in 2000, in my experience, Linux took over from Solaris/SunOS because it was free compared to lots of $$ for Sun and PCs cheaper than Sparc boxes.
But that's MY perception of that period of time. I intentionally avoided working for Windows-based places because it sucked so bad.
Re:Microsoft blames monster email outage on software defect ...
by thwarted on Monday November 16, 2026 @10:00PM
(#50526848)
> Seems like a stretch TO ME to say Windows was ever the dominant player in any major server market - SunOS/Solaris was by far the biggest server platform for startups / dotcom boom from say 1995-2002
I agree. We're talking web servers here, not SMB file and print servers, which were mostly WindowsNT based and the contrasting vendor would be Novell until sometime in the late 90s.
The vendor of popular web server software over time is well documented thanks to 30 years of Netcraft. Unfortunately, we don't see the platform, but most uses of Apache were either Linux or Sun based, and Microsoft shops tended to use IIS and rarely anything open source. My own experience at the time was that things that ran on Sun was mostly java based, WebLogic.
Microsoft's share of the top 1 million busiest sites was never above 20% and has been steadily declining for the last 15 years. If you were serving a lot of traffic, you weren't (and aren't) using Windows to do it. And the majority of traffic has always been served by something other than Microsoft.
Microsoft reached over 50% in 2017 in marketshare of all sites, but traditionally has been limited to below 40% and has really dropped off since 2020. And (based on the graphs), that traffic probably migrated to nginx.
There was a period in here where Netcraft had to make a distinction between sites and servers. Cheapo web hosting services via virtual hosting was on the rise, and a massive number of domains/sites were all served from the same server, and it skewed the numbers in terms of vendor market penetration (think shared-hosting offerings like GoDaddy, the hosting industry was lousy with them). The back and forth churn was notable when some big hosting company switched their web server software and then hundreds of thousands of sites changed software vendors. I remember Microsoft making a big deal about this with marketing copy like "The most popular web server software on the Internet", which may have been true based on domains/sites, but not based on installed instances or most-served-traffic.
IME Microsoft was never a big vendor of web server software, unlike how they are a big vendor of desktop. People were aware of IIS as an option if you were a Microsoft shop, and I know some people who ran it (I once crashed my buddy's site using the perl WinNuke one-liner to put him in his place after bragging about his NT and IIS setup -- hi Noel!) but Sun dominated and where it didn't, it was open source solutions.
> I intentionally avoided working for Windows-based places because it sucked so bad.
Re:Microsoft blames monster email outage on software defect ...
by jamesfinlayson on Monday November 16, 2026 @08:54PM
(#50526406)
Yeah a place I was at rebuilt their ancient mainframe system in the mid 1990s - being a "big businessy" company they chose Windows (I assume they would have been using other Microsoft products at the time so in that sense, going with something else would have made less sense).
Re:Microsoft blames monster email outage on software defect ...
by MiroslavPokorny on Tuesday November 17, 2026 @02:36AM
(#50528561)
Something as big as Windows is not the product of a single or a few people, but hundreds.
The problem is making all the moving parts from all those people work together well. Weakest link is almost never the people but the complexity of everything.
Re:Microsoft blames monster email outage on software defect ...
by MiroslavPokorny on Tuesday November 17, 2026 @07:42PM
(#50540364)
Exactly its prolly tens of thousands past and present. Its basically impossible for a single person to understand the ramifications of changing any large architecture feature.
Re:Microsoft blames monster email outage on software defect ...
by skydhash on Monday November 16, 2026 @07:52PM
(#50525927)
It's a nice engine, in a very cumbersome bulldozer, while people just want a small sedan to go to the store. Linux is the worse kernel, but it's simpler to use.
Re:Microsoft blames monster email outage on software defect ...
by devy on Monday November 16, 2026 @05:17PM
(#50524202)
Interesting that it's their EVP of Copilot, Agents, and Platform at Microsoft, building the next generation of AI-powered tools for work who spoke up for he post-mortem instead of their CTO.
Does that mean Microsoft's internal code are 100% outsourced to agents / AI and human review just won't be able to catch latent bugs like this?
Re:Microsoft blames monster email outage on software defect ...
by syngrog66 on Monday November 16, 2026 @05:48PM
(#50524602)
broadstrokes: bitten by concurrency, contention, thundering herd, and a proximate cause which was not in fact the root cause and the root cause had been lurking latent for several months so hard to undo surgically fast. cobined with insufficient load/release testing and insufficient mitigation in prod.
Re:Microsoft blames monster email outage on software defect ...
by andix on Monday November 16, 2026 @07:48PM
(#50525888)
There is one thing I don't get with this issue. The problem was basically that their system couldn't handle the normal load. Why couldn't they just reduce the load at some point just by dropping/limiting connections?
For example just accept connections from specific (random) IP ranges. They could even cycle through different ranges every 5 minutes to give clients a chance to sync at least once every hour.
The reduced load could be used to gradually bring up servers again without failing, add additional capacity, and gradually restoring access. Completely without fixing the bug at all. The system managed to handle usual load even with the bug, it just hit some critical point that made it collapse. Adding more resources would also fix that.
Re:Microsoft blames monster email outage on software defect ...
by itopaloglu83 on Monday November 16, 2026 @08:37PM
(#50526283)
You mean they should've made disaster recovery plans ahead of time? That requires experience and a certain level of know-how, in a company that has just decided to "voluntarily retire” a quarter of its most experienced workforce. There was a lot of things they could've, should've, and would've done.
Re:Microsoft blames monster email outage on software defect ...
by andix on Monday November 16, 2026 @08:56PM
(#50526419)
I came up with another theory: maybe it was internal load, server to server communication. For example email traffic between Outlook mailboxes (or some kind of API calls). Once the queues were filled up, every server added back to the pool just increased internal load by hammering other servers.
Restricting external access wouldn't solve this issue.
I guess adding a config option for throttling queue processing is something any distributed system should have.
Re:Microsoft blames monster email outage on software defect ...
by mike_hearn on Tuesday November 17, 2026 @05:05AM
(#50529521)
The issue appears to have been in one of the load balancer layers themselves (the "front door"). So that makes it a lot harder.
Most big cloud services operate several layers of load balancing. The first is a smart DNS server that turns a lookup of mail.outlook.com or whatever into an IP address of a datacenter roughly nearby to where the user is, to a datacenter that's not too overloaded. But because DNS lookups are cached and sometimes cached too long by clients this is a very rough and ready form of LB.
The second layer are routers that spread requests to a single IP address over "hardware" LBs that take raw TCP/IP connections from the internet. Then those machines do some basic health checking and forwarding to the frontend web servers. At that level the machines aren't aware of what service or customer the connections are for, they're just doing some flow hashing to pick a "backend" (another frontend).
Then the cloud/service level machines terminate the TCP/IP connection and often TLS as well. They're reverse proxies that do URL matching and forward traffic on to the app specific servers. This is the first layer at which you can start to intelligently inspect or drop traffic.
No one person in a big org has the ability to limit traffic to a service across every layer of load balancing that exists. If the reverse proxies start dying, reducing traffic would mean changing the configuration of the hardware LBs which is usually a different team.
Re:Microsoft blames monster email outage on software defect ...
by filleokus on Tuesday November 17, 2026 @08:03AM
(#50530808)
But given the size of the impact this incident had, I assume involving any team that would not be an issue.
I'm guessing there must be some deeper issue at play making it too risky or not possible. Maybe the hardware LB's were too intertwined with customer Azure loads or something?
But when you are hitting hour 12 of a global outage of one of your most central services I would assume even quite exotic solutions would be considered (e.g standing up dedicated hardware LB's with load shedding and pointing some of the DNS resolution to those)
Re:Microsoft blames monster email outage on software defect ...
by agentphil on Wednesday November 18, 2026 @12:51AM
(#50542349)
> The first is a smart DNS server that turns a lookup of mail.outlook.com or whatever into an IP address of a datacenter roughly nearby to where the user is, to a datacenter that's not too overloaded. But because DNS lookups are cached and sometimes cached too long by clients this is a very rough and ready form of LB.
This looks like a potential weak point against a cyber attack. A big number of distributed bots in one geographical area can send their requests to one data center and overload it.
Re:Microsoft blames monster email outage on software defect ...
by mike_hearn on Wednesday November 18, 2026 @03:59AM
(#50543740)
That's actually easy to solve. There was an attack exactly like that when I worked at Google (a loooong time ago). The DDoS bots cached the DNS lookup forever, but compliant clients respect the 5 min TTL. So the cluster was drained at the DNS level. User traffic went elsewhere and the botnet DDoSd an empty cluster nobody was using. Attack was defended against perfectly.
Re:Microsoft blames monster email outage on software defect ...
by d_g_m on Monday November 16, 2026 @10:49PM
(#50527198)
> "Our load testing tended to focus on weaker hardware with fewer cores....The contention issue is worsened on machines with more cores and newer hardware."
Does anyone know why the situation got worse on better machines instead of worse? Normally it is the other way around when it comes to test coverage of hardware specs
Re:Microsoft blames monster email outage on software defect ...
by nl on Monday November 16, 2026 @11:05PM
(#50527288)
From the report: "lock contention inside an internal process... The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores."
Re:Microsoft blames monster email outage on software defect ...
by tremon on Tuesday November 17, 2026 @11:20AM
(#50533665)
Lock contention is when multiple threads try to access the same (memory) resource. More cores == more threads accessing the resource, and faster execution == less time between resource access for each thread. Both factors increase the time spent waiting on said lock, in relative and in absolute terms.
Re:Microsoft blames monster email outage on software defect ...
by hinkley on Tuesday November 17, 2026 @05:37AM
(#50529715)
The cost of the outage won’t be accounted for years. They refunded a little money to big customers on losses that could be in the millions. Some customers will leave. More will leave the next time this happens. And it would be disingenuous to blame the next failure for the customer turnover because it will be the last straw for those who leave. Not the only straw.
Re:Microsoft blames monster email outage on software defect ...
by 306bobby on Tuesday November 17, 2026 @08:59AM
(#50531461)
I have a self hosted email server. IP is not blacklisted, has decent reputation, and I pass every single test I've come across for validation and delivery
I will still get sent to junk 70% of the time, especially to Microsoft Exchange
Re:Microsoft blames monster email outage on software defect ...
by vladvasiliu on Tuesday November 17, 2026 @10:02AM
(#50532313)
To be fair, most e-mails coming from MS themselves to my work e-mail, hosted on o365, end up in junk.
I'm an IT admin at my company (though I don't often actually admin the MS parts), so I regularly receive e-mails from [email protected] that almost always end up in the junk folder. The mail is legitimate. But it's typically about such-and-such doing something that's against some policy, so we get notifications. Since I'm not in charge of either side of the policy violation, I could never be bothered to look into why these messages end up in the junk folder.
Re:Microsoft blames monster email outage on software defect ...
by tedeh on Tuesday November 17, 2026 @03:28AM
(#50528879)
No one has asked it because the only possible reply is uninformative PR-slop. Even if AI caused this, which I'm not saying, that could not be admitted because AI is the New Big Thing and Microslop is like an anxious school girl and they don't want to be seen to be left behind/don't know what they are doing, because that would be bad for their social standing, sorry, stock price.
Re:Microsoft blames monster email outage on software defect ...
by quikoa on Tuesday November 17, 2026 @03:41AM
(#50528977)
Microsoft would never admit to AI being at fault. Businesses praise AI if something goes right and blame people if something goes wrong. If the opposite happens to be true then they'll simply won't share what caused it.
Re:Microsoft blames monster email outage on software defect ...
by wrasee on Tuesday November 17, 2026 @06:33AM
(#50530072)
What’s interesting is what they did admit to, that this was introduced in a change made in July 2026. They didn’t have to mention that specifically, and that date alone is carrying a lot of indirect information.
Re:Microsoft blames monster email outage on software defect ...
by frereubu on Tuesday November 17, 2026 @05:34AM
(#50529699)
Sibling comments are right that Microsoft would never admit that publicly, but I do think it's worth asking on HN in case there are MS employees who might be able to create a burner account and respond without revealing their identity. Of course it's difficult to verify whether what a burner account says is true, but I still think it would be of interest.
Re:Microsoft blames monster email outage on software defect ...
by servo_sausage on Tuesday November 17, 2026 @06:57AM
(#50530261)
Similar things come along in outsourcing/low costing discussion. The software gets worse, the people who should be highly paid and retained are replaced by systems or people that can only cope with the normal 99%, and management thinks they are getting a much better deal.
Re:Microsoft blames monster email outage on software defect ...
by renegade-otter on Tuesday November 17, 2026 @09:17AM
(#50531713)
The thing about AI is that it does find exotic edge cases in your code. Regular production code evolves by hitting those edge cases. It slowly matures.
Seeing AI "fix" the code makes everyone just turn off the brains and move on.
Those here who program know - no one reads code or PRS.
Re:Microsoft blames monster email outage on software defect ...
by renegade-otter on Tuesday November 17, 2026 @09:27AM
(#50531857)
The thing about AI is that it does find exotic edge cases in your code. Regular production code evolves by hitting those edge cases. It slowly matures.
Seeing AI "fix" the code makes everyone just turn off the brains and move on.
Those here who program know - no one reads code or PRs.
Re:Microsoft blames monster email outage on software defect ...
by smallstepforman on Tuesday November 17, 2026 @03:29AM
(#50528893)
Monoculture is still a problem, from biology to markets to computers. People look at me puzzingly when they hear I run Haiku, that I write my own rendering engine, that I’m building a small country estate.
Re:Microsoft blames monster email outage on software defect ...
by malkia on Tuesday November 17, 2026 @12:08PM
(#50534559)
Lock contention, especially cache line contention in large NUMA systems. If hardware was upgraded to NUMA machines, or VM's were not cordoned each to sit in each it's own NUMA node (CPU, Caches & Memory) then something like this would happen.
We saw with our own tools back when an artist were given a low-core "server" like machine with many CPUs (good we thought), and 2 NUMA nodes, and suddely the level editor for our game was running much slower. We could blame std::shared_ptr<> all day long, or similar but it's there and it was real.
Tried the 4kb interleaved paging, but then everything was equally slow. At the end we simply blocked the App to run only on one NUMA node (still using memory from the other one), and things were back to "normal". "Normal", because promises were not kept - we could not use all CPUs.
Re:Microsoft blames monster email outage on software defect ...
by kozikow on Tuesday November 17, 2026 @03:22PM
(#50537636)
Of course it's easy to say "haha AI slop".
Stuff like this also happened prior to AI. One of reasons it happens more frequently now is just now timelines of product development are compressed. So let's say we get 3x outages, but at 10x the speed of new features.
To me it sounds like Outlook team do not have proper gradual rollout for service of this size. Like rollout the new version to 10% servers, let it soak, compare metrics before rolling up higher. Most of "bigtechs" have something like this nowadays.
Is this commonly acceptable? I'd imagine it's a costly trade-off?
But if low is 64 and high is a whole cluster, now that's a different kind of maths.
We solved that by sharding customers, so about every 1000 database clusters got their own k8s cluster; production then could run hundreds of those “orchestras”, but we could afford to load test one of them and be relatively confident the test was valid for production.
This relied on the various cloud vendors running the front-line load balancers that took traffic and moved it into the shards though. I was about to say that seemed to be the case for Microsoft here as well, but actually it seems they load tested an individual server with different specs than prod.
Tricky problem though - it seems they reasoned if a lower specced server passed the load test, a much beefier prod server would handle the load too. Wrong, but you can see why someone made that assumption.
I doubt this very much.
The load testing env is the Office 365 load testing environment. I'd be surprised if it is less than 10,000 cores, and the machines are probably good but older; possibly 50 core Xeons instead of 128 core EPYC or something.
It seems likely that the test env was setup to mirror one of the smaller 365 regions a few years ago and the only updates were when machines were replaced.
I've seen NUMA cache line contention affecting us for real (on Windows, for our game tools). Limiting certain tools to run locked under one NUMA node helped a lot. It's real.
The updated CoreInfo https://learn.microsoft.com/en-us/sysinternals/downloads/cor... shows this well
plus, mistakes are the essence of learning and sometimes mistakes are necessary.
likewise, we are a very efficient world and efficiency hates redundancy, and hates nonconformity...of which those are two of the features that make exploits widely successful and failures massively impactful
We need to teach queuing theory in school. Intuition about how much redundancy you can squeeze out if a system and still handle hiccups is woefully inaccurate. So we under provision and are surprised when traffic jams take hours and hours to clear. Despite that being completely predictable.
If you get 990 with a capacity of 1000, it takes 99 minutes to clear that same disruption. And if you get 1000 with a capacity of 1000, average time to recover from a little random fluctuation is infinity.
I believe normal practice is to upgrade fiber-optic lines when they're somewhere between 50% and 80% utilisation.
Also “9 minutes to clear” only tells how long the congestion is, NOT what the customer experiences. On a typical response time of .6s, after the delay the first customer sees 101x the usual response time tapering off to 2x almost 9 minutes later. That is a very different story than, “things will be slow for 9 minutes”. It’s everything will be an absolute shitshow for 9 minutes.
Professor made a simple statement, pointed to the local (I405) freeway in Irvine and stated that traffic is incompressible flow similar to water. Flow problems are only solved by a bigger pipe and whenever you see its manifestations (flow memory, slowing at chokepoints, etc.) remember that fact. Also remember: there will always be a single slowest point in a network unless you can somehow manage to cause the flow to accelerate past and away from obstructions.
The first guy I knew who was write-a-book invested in Agile was working out Fluid Dynamics analogies with me when we got talking shop. We stopped working together before I made the queuing theory connection.
Microsoft tests its stuff on their own employees (old citation, but still true today: https://devblogs.microsoft.com/oldnewthing/20110802-00/?p=10...), and that includes Exchange online. However, even though Microsoft is a pretty big company, it's still teeny compared to its customer base.
They find QA dirty in their statement, because they don't state who set that priority, nor why the new hardware was felt to be less likely to be affected (what did their risk matrix look like for their hardware profiles i wonder)
You mean you and me? You realize pensions, institutions, and individual investors are shareholders as well?
The mantra of "shareholder primacy" sounds great in theory but just becomes an excuse for short term profit boosts for a small class of shareholders usually at the expensive of the long term health and growth of the company and long term value seeking shareholders.
Sort of similar to how most countries with "people's" or "democratic" in the government's name tend to be democratic only for the ruling party.
In this case ensuring adequate engineering resources for possibly the most widely used email system in the world doesn't seem crazy. How many long term customers will they loose because of this incident? Perhaps they did have enough resources, but given the recent layoffs to chase AI I sort of doubt it.
Not for me and probably a lot of people and institutions too. We have retirement accounts, endowments, and pensions tied to index funds and the index funds have these stocks in them.
We need these companies to do well to ensure we retire and grow well.
Workers benefit much more from higher wages than higher stock prices. Simply because much more of their income comes from wages. So whenever someone says "Think about the retirees and other ordinary shareholders" in response to a suggestion to pay workers more than absolutely necessary, they're full of shit.
This "rich people will benefit so it's automatically bad" mentality is counterproductive and borne out of jealousy.
If part of your retirement income comes from another source, such as rental property, workers being paid more is better for you because they can afford higher rents.
Ordinary workers would better off if they were simply paid more and could put more money into their retirement savings.
We all need to side with whoever makes the charts go up and to right to ensure our post-retirement lives are well funded.
Myself and many like me don't care what side that is and we don't care much for policies, groups, or programs that are counter to that.
For at least the first 11 years of most people's careers, contributions outweigh gains. And only if they continuously contribute the same amount every month.
Being paid more, especially early career, is just more useful to the 99% white collar workers on HN. It lets you hit the crossover point faster. It lets you save more money for retirement. Compounding more money earlier gives you a better retirement. This is simple math. It isn't a class conflict argument.
https://finance.yahoo.com/news/crossover-point-compounding-m...
All me and the other guy are saying is that it's possible to not be filthy rich and still want stocks to do well. "Think of the poor starving shareholders" implies that we commoners shouldn't care or want that. But that's false. I'm no longer working, and have no rental income, so I need my investments to do well.
That necessarily means lower profits to shareholders. The two goals are in tension with each other.
> I'm no longer working, and have no rental income, so I need my investments to do well.
Then you're still rich. Not a 1%-er but very possibly a 10%-er. I like my investments to do well too. I don't pretend it's in the common good.
The fact that you can't apply logical reasoning is your own limitation and you shouldn't presume it applies broadly to everyone.
For example in my country the previous extreme right wing government lowered taxes to high earners. That includes me. I did pay slightly less in taxes, while people who make much more money than me had a much bigger reduction.
So I also had an advantage? LOL NO.
I did not because they also doubled the cost to go and see a doctor so my slight tax reduction wasn't even enough to pay for that.
Of course people who can't do math think "uh good less tax! I vote for them again!", but people who can do some 3rd grade mathematics might realise that they actually paid more.
And we aren't even including the increased health risk in going around when the less wealthy people will naturally tend to go less to the doctor because they are being priced out and infect everyone.
I like ragging on Microsoft as much as the next guy, but honestly this problem sounds complex & hard to predict. AI or not, this feels like the type of bug that would have slipped by me as well.
The response clearly could have been stronger though.
The opposite is true running in production for the last several months. They were runnign on big production machines with enough cores, but never quite had enough load to trigger the lock contention. But once both conditions were true, big machines and high load, due to some unrelated failover elsewhere suddently bringing more load, we then had the magic combination.
It's really not unreasonable to expect an enormous hyperscaler like Microsoft to do load testing on hardware that exactly matches their production systems.
Github copilot had INSANE usage. Cursor switched away from per request plans in 2025, Copilot did it this year with all the reasoning and more expensive models. It was insane, you could write out a whole page of features, and have it run on one request for hours depending on the loopholes you exploited (some models were discounted for some time, basically free in some cases, and shenanigans around switching in betwen agents)
We talk about opus et. al being heavily subsidized at like 1-2k for $30-$100. I remember seeing figures in the 10s of thousands in subsidizations for GHC. There's no way they weren't losing just massive piles of money.
What does this mean? I didn't understand
In a rare occurrence, The Register sides with the Microsoft developers. Not that they're anti-developers, but they are anti Microsoft because of its well-earned reputation for EEE and Halloween Documents.
On a qualifying purchase of at minimum $50. Terms and conditions apply.
That's not to say their job was or is easy: nursing is far more hazardous than SWE, and significantly more physically and mentally demanding, too. But I do think many got more compensation than your $5 gift certificate implies.
But as a software engineer, if I work overtime on an outage like the above, I'll get precisely $0 of additional pay for it.
If your company does not have Overtime policy, it's a shitty one.
(Not that, however — having worked many times in the middle of the night — that I disagree with your moral assertions about that.)
Surprised so many people are struggling with this...
Having been on incidents like this before though, yea, that aint happening.
Yeah, instead what will happen is their manager will ask them why they didn't get as many story points done this sprint as they did last sprint and their performance will be marked "degraded".
Been there, through that, and sadly you're right. The bitter part is I didn't want extra compensation or anything. Just a thank you and a day off is more than enough for me, so to recharge and ride the wave of what we (not I, we) have accomplished, so I can work with extra motivation for some time.
Instead, got a 5 minute coffee break and dived to the next non-urgent ticket. Yeah, great way to thank and motivate the people who spent 24 hours in an underground office cutoff from the world, trying to save a system from melting down.
Glad I'm not there anymore.
https://www.geekwire.com/2026/microsoft-makes-new-voluntary-...
How long until Microsoft fires key people to the point that outages become more common? All in the name of AI, which I don't believe for a second, they're just trying to maintain stock market profitability math by firing enough, they'll just hire more people when they make more.
Already happening. Granted, not a global outage like this one, but there's not a single day without a service degradation or advisory on the admin portal for 365. The "It's now called Microsoft 360" memes are well earned, and their support is atrocious. I get the feeling that everyone who knows how it works is gone.
SOMETIMES the company was being prepared for sale...THEN you can only hope you get let go in the first few rounds BEFORE the money for severances runs out
I am sorry that somehow society is beginning to sell the idea that the American workforce does not or need not include rough and ready individuals.
I am aged out of the workforce at this point and find it hard to re-enter due to different problems
I know I shouldn't be surprised, but I still struggle to imagine such high-capacity servers running with windows. Part of me wonders why any important server would be running windows. And I do chuckle at the "update" that returned windows to a previous state. Never admit a rollback. Never admit a previous update was less than perfect. Hold the line MS. Hold that line.
Here, I converted it to corporate speak for you.
Microsoft does greenfield new stuff on Linux mostly.
Moreover, my current switch to Linux shows that Linux is not so much better in stability as it is in public imagination. Once a month or two, I have KDE freeze so hard I barely can recover without reset (sometimes can't). Still can't find the reason. Hardware seems to be okay, it is an expensive gaming laptop.
Also, out of RAM can lead to GUI crashing even with swap enabled. True, this is unrelated to servers, which run without GUI, but still weird that it happens.
Also, when my laptop runs for weeks without reboot, it has issues with VRAM being occupied and not freed for some reason, which causes game crashes. With all the talk about Linux infinite uptime I expected better.
However, I almost can't complain about my homelab server. If there is enough disk space, it can run for months. Unfortunately, lately there are so many patches, I need to reboot almost each day (at least podman-hosting VM, if not proxmox host)
Windows 10?
7?
XP?
95?
Indeed they do deserve a long hard look.
Toss1's conjecture:
We are now seeing the beginning of an era of latent technical debt brought about by too rapidly firing developers and switching to AI code generation (nevermind vibe-coding). Even well-prompted and well-monitored by top expert coders and architects, there is latent technical debt, and it is more diffuse, harder to find, and harder to fix, and will start showing up in outages like this.
(It first seemed the ATC outage was another example, but it now seems it was merely the usual management incompetence if not an outright manipulative ploy to 'justify' more spending for a single-bid contract [0])
[0] https://ajsignalnotnoise.substack.com/p/faa-air-traffic-cont...
- our testing didn't cover it (again, this is expected occasionally).
- this cascaded to take down more than a billion people's service (WTF; this is not how you design systems).
- it took 15 hours to deploy the fix (not good, but at this scale maybe).
Feels like they are completely missing the point of where the problem is and how to fix it?
So, even when you fix the issue, it takes a bit to clear up, and the load gauge stays at the red zone for a while.
This lets you start at a very small amount of load and gradually ramp up. But you can’t reject randomly or goodput stays at zero and the backlog never decreases.
I believe SDN systems can do that with ease, but what would be the equivalent method on older systems? Selectively allowing IPs to go through the trickle load into the system?
Distribution of users by /8 isn't very even and the order that customers will get online wouldn't be an order anyone would pick, but it should be reasonable to implement and synchronize (if needed) and it lets you make progress.
Another idea is to have different clients use different ports (enterprise customers on one port, SMB on another) and turn up in the order of SLA.
I've definitely been in similar scale war rooms and 15 hours is not acceptable.
The thing with plain outages vs. say, partial degradation of services, you can roll out fast because you're already at worst case scenario *[1].
Here's an interesting one to compare to that I'm very familiar with:
https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbn...
Another really bad one I did not have any involvement in but studied extensively post hoc:
https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
*[1] - I suppose data loss is worse but let's assume most services are stateless.
What's not acceptable to me is a bug of this impact going undetected for 6+ months in production.
You can't even say "but managing fleet/cells is harder": they make software for that, too!
But at the end of the day these are servers strung together sending packets over copper wires. They are extremely reliant till they are not.
When crisis strikes you can only hope there is a squad of well trained humans who can step in to untangle these messes. But as more of the labor force is stripped out of responsibilities, the longer these outages will take to resolve. Imagine being jolted out of your year-long UBI-funded vacation to login and resolve a multi-continent server outage that you only have a vague memory of.
That’s Silicon Valley fantasy, it’s not even close to reality
Cynic in me does not see any need to change anything.
The amount of time, energy, money it would take to have an objectively bad fallback option is crazy.
And then what? Most of the rest of the world is still down...
If I go with Microsoft and they have an outage, everyone is down and we can all commiserate about how it’s Microsoft’s fault and outside our control.
That only scales until stuff takes 2 weeks to come up again. That already happened for us with Atlassian. That was our fault as well, apparently.
* companies should have multiple parallel email and login systems (for everyone so they can switch when one is down)
* not all companies should use Microsoft hosted login and email services
But both aren't very convincing. Having parallel login systems seems like an awful idea and parallel email systems like a very expensive project with little upside (switching email systems on dns seems fun).
And most companies are already not on Microsoft, just a lot are. (it is also not obvious how that would improve the situation for the companies)
And it's also possible that these normal accidents might become more frequent with all the layoffs, AI code, and AI-generated exploits. I guess we'll see.
The electricity industry has this notion they call a black start, where they bring the grid online from zero. It's a very difficult operation because you can't start a generator straight into an offline grid otherwise it gets hit with full demand and can't keep frequency in the right range, which can damage equipment. So they have to start by disconnecting all the loads and then bringing load blocks back in sequence, trying to roughly balance available generation with the size of the next block. In practice black starts are extremely rare, but they can happen after a cascading failure in which something causes some generating stations to trip, redirecting load onto the remaining stations which can't handle it causing them to trip in turn. The entire grid can collapse in just 60 seconds or so once that kind of runaway failure starts.
This kind of outage is very analogous to a black start. A few servers get into trouble - maybe they're in a crash loop or maybe their caches are cold - the load balancers shift traffic onto the remainder which then can't handle the load either and so the failure cascades. Once a service is in this state sometimes the only remedy is to disconnect all load, bring the servers back to stability, then bring load blocks back online in small chunks, making sure to never hit anything with the full load immediately to ensure any caches or JIT compilers have time to warm up.
Black starting a service is obviously hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster. I used to be an SRE at Google and although we regularly started up new clusters and moved traffic around between them, actually starting an isolated service instance entirely from scratch was never something we practiced. I remember in a few cases teams discovered devs had introduced circular dependencies by accident, i.e. starting service X required service X to already exist, or it depended on service Y that in turn depended on X. Sometimes you can only find these things by practicing.
I did SRE at another firm, not Google like the parent but a similar level. We tested these procedures twice a year and every time we added a region.
I agree: no excuse.
SBB just finished their project to be able to restart the grid in 5min. (There was a famous multi hour outage a few decade ago that I think motivated this work)
If the outage involves damage to transformers then all bets are off. In those cases some areas of the country might become uninhabitable for months.
A power grid is different because it's not about the power immediately being used all at once, it's about stabilizing the frequency. Turning generators on takes a while to stabilize, each reactor may output inconsistently for a while, which is why you never start a generator and immediately ask for full output, you start it, let it warm up, then add load. Which does sound like a server, but you don't need every server speaking the same frequency.
If a grids frequency is off, or stutters it kills electronics. If a servers JIT cache is ready, it's slow. The extremes are completely different.
Now the same garden hose problem exists, going from 0 to everyone logging in at once is a lot of load. But that's why you come back online with more servers and scale back.
It went remarkably well.
The initial start up (of the cold start plants) took the double of time expected. The "cold start" capacity is being increased due to some of the issues found.
Portugal was able to recover in 12h (still a lot) without any external assistance.
> By 07:00 CEST (05:00 UTC) on 29 April, electricity had been restored to 99% of energy demand, and full restoration was achieved by 11:00 CEST (09:00 UTC) that same day.
The wikipedia references seem valid for this precise timeline, which is more informative than your remark.
https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou...
I'm not trying to win a pissing match, just trying to counter the very common miserabilist notion that Portugal did worse than Spain.
This specific problem was (1) caused in Spain, (2) fixed faster in Portugal, and (3) without external help.
Portugal could've been the cause, and could've done better, for sure.
But it's just not balanced to write "In Portugal it didn't go that well" as a response to "It went remarkably well [in Spain]." Makes it sound much worse than it actually was.
Do you have any engineering insight on why a hydropower plant was used, and not, say, a gas-fired power plant?
Once the hydro station is back online the power it generates can be used to provide house load to the other stations.
But load management while bootstrapping the rest of the grid is still very hard. Often the reason for really long blackout, ones measured in days rather than hours, is repeated cascading failures as operators try to reconnect parts of a the grid, and accidentally overload recovered parts of the grid, causing generators to trip off again.
Typically <5% afaik
They should have that and they typically pay substantially more for it. Unfortunately it's also the case that the private companies providing said black-start capability may have just not bothered to tell anyone they haven't been able to do it for a few months\years.
A properly run network will require those facilities to be tested multiple times a year to ensure it actually works, and make payment for the facility contingent on passing those tests.
So you need a hydro plant that is either small enough that it’s using turbines with at least some permanent magnets (you get turbine varieties that basically have a mini turbine inside using magnets to power the energise the larger electromagnets), or has dedicated infra to produce power to energise the coils in their turbines before they’re producing enough power to energise themselves.
The process is called islanding.
This is very much like trying to jump onto a train from another train moving alongside it.
We had to practice load pickup on a normal start at the pier often, but we likely practiced a full SCRAM to back online (including a FULL load shutdown) more.
Having predefined lists of what gets brought up at various points helps a ton, but it's still not an easy or simple operation.
[1] https://www.youtube.com/watch?v=uOSnQM1Zu4w
You need a temporary source of power to start the water pumps and it takes a while for the water to flow all the way through the system and for the coal to saturate all the belts and power plants, if you hook it up to the main grid before all the plants are operational then it shuts down again.
Factorio makes this fairly easy for the most part, a cold start isn't really some massive engineering challenge, but still, worth guarding against. By the time you get solar and / or nuclear power is basically solved, at least on Nauvis, but it's "overengineering the game," so... we overengineer.
In Factorio, solar is an easy source of cold start. In the real world, I think hydropower is a commonly designated choice and the solar farms have to wait for a grid to sync against.
Cold starts are a constant in Aquilo until you research and build fusion power. And whatever you do by hand to handle them, there's a device in the game that does it automatically too.
Once we’ve restored power to the substations, we can carry out what are known as voltage restoration operations.
At a nuclear power station, if all goes well, there will be at least one nuclear reactor unit that has managed to operate in island mode. They have their own sub station and can restart the others units.
We restart as many units as possible on a virtually empty section of the grid (serving only essential and vital consumers such as hospitals). We then reconnect consumers in a predefined order to avoid significant fluctuations that could trigger further cascading failures.
The most difficult part is often identifying the exact cause of the blackout with certainty, so as to prevent it from repeating it.
“Ready? 10 seconds to power on server alpha-romero-one-o-four!”
“Load—at five percent, check! All systems stable!”
It's chaotic and stressful in a slow asynchronous way.
"Rabbit, flu shot, someone talk to me!"
(I have no experience with H1Bs nor beef, I'm a German who's merely had the joy of seeing failed offshore projects.)
All of it is documented through a unified catalog of services and their associated metadata.
Source: Worked in MSR and shipped services through Azure / CoreAI.
I mean... you can try, but unless you spend a lot of resources on it, I don't expect test load to be close enough to really practice a black start.
From experience, the best way to prepare for restoration from nasty outages is to have them frequently! The team I was on was pretty good at outage restoration, until we started regularly hitting about a year between outages. After a few years of one or maybe two outages per year, we had lost the muscle memory for protecting services that get overloaded during or after an outage and setting outage messages (which both informs customers and protects customer service queues).
Regularly starting up an isolated cluster at least lets you get ahead of sequencing issues though; hopefully it's enough if you can get the whole cluster started and then modulate inbound traffic (most complex services will need something like you mentioned so that you limit the number of sessions, but sessions that start are likely to proceed successfully).
This is one of those beautiful engineer understatements. Rotational speed is fundamentally linked to voltage by the motor velocity constant: https://en.wikipedia.org/wiki/Motor_constants
Although it's fundamental for all motors, the actual constant is arbitrary for each motor. Voltage is still proportional to change in field is proportional to speed; at zero voltage you have zero speed.
A large dead AC load has its own speed- zero. It's connecting the engine without a clutch, but the engine makes hundreds of megawatts. The 6' wide generator wants to go from 3600 RPM to a standstill in a few inches. A full speed rotor can have up to 5 gigajoules of rotational inertia; over a ton of TNT.
On a live grid you've got a synchronoscope: https://youtu.be/LVozbX9Mn1c?si=RhRPIkuTVxh4Wseq&t=181
Which is great, because connecting it backwards would cause a slow motion explosion. On a dead grid the only thing saving you is the resistance in the wires; it would be LOUD if you connected to it and you would very likely increase the ventilation through the walls of the plant.
[0] https://youtu.be/xGQxSJmadm0?si=fVdJXtN0bCBX_fyW
That we often bypass local field control and tie large industrial synchronous and asynchronous (induction) motors directly to grid is narrow and quirky special case that is common for historic raisins but really quite awful, with staged delta-star or dahlander winding relay tricks to go from insane start currents down to just absurd.
Brushed motors are just synchronous motors with a primitive mechanical field control attached, allowing them to reduce the field rate in response to load.
How long do you need to dry a grape for it to not only become a raisin but a historic one?
In a synchronised power grid, machine speed is connected to electrical frequency, that's it.
Voltage and reactive power are controlled by the strength of the rotating field, not speed.
(It might be better not to use CIDR ranges, but use the lower bits instead if possible. For example starting to unblock all IPs that end with .80 until .90, to get more random geographical distribution)
Which leads to my suspicion it was internal load that brought the system down. Communication between the servers (like delivering queued up messages from one O365 account to another), and not external access from users and apps.
If the load balancer already does some connection tracking, it can just keep serving existing connections and only drop new ones. If connection tracking is not available, the IP blocklist would be a suitable option.
IP blocking is generally not what you want though.
Okay, they get different IPs during the day, fine. But it's still kind of randomized. Stepping through IP ranges will reduce load and give access to more people.
I really don't get the issue you are seeing with this.
IIRC Microsoft posted a comment around 10pm where they discussed rate-limiting external connections, and many reactions on last week's HN thread were like "wait, you so didn't even drop load before now?" So internal load may have been a contributing factor, but it's by no means certain given that they didn't stop serving clients.
This is absolutely terrifying knowledge to know given the competency of the current administration and their attitude/response to recent "black swan" outages.
Readers who know about distributed systems and their failure modes already understand the thing being explained. They (and you) are not the target audience. Let people who are not already experts learn something by analogy.
But the analogy is useful in another way. The electricity industry plans and rehearses for this kind of thing, even though it's impossible to simulate. I used to be rather skeptical that these rehearsals would be effective but Portugal black started last year after a solar plant in Spain malfunctioned and took down the whole Iberian grid. It took almost exactly 24 hours despite the grid collapsing several times during the process, which is the usual estimate for how long things will take. So it seems they have this stuff well under control.
I won't slam Microsoft too hard for this outage because this type of scenario has played out at many companies over the years. Yet I never encountered an ops team that has written procedures for how to recover when the entire service has entered a crash loop under peak load.
The exact analogy we use for this doesn't matter much as they are all pretty evocative, but using black start has the advantage of reminding us that there's another branch of engineering where this necessity is taken seriously and rehearsed. They also engineer for it ahead of time so they can prioritize customers like hospitals and airports under overload conditions. I think it would be a good feature for cloud LBs to be routinely differentiating between e.g. customers on paid plans vs free so they can serve paid customers first if there aren't enough live backends.
this is an interesting observation, since this should be table stakes for running a high scale distributed system. there are plenty of companies that do this and write about this extensively. it definitely becomes less important until you get to certain levels of scale though - a random app with 10k users has a different standard than, e.g., google cloud storage.
At the time Iberia was relying >50% on renewables which is very dangerous. The operators tried to recall gas power plants to provide inertial stability, but the plants were cold as they'd been taken offline in the morning to make way for solar (prioritized due to green politics), and even though gas plants can spin up very fast compared to other kinds they could not warm up fast enough to rescue the situation as it was all over thirty seconds later.
So it depends how you define "fault" but yes the starting point was a broken solar plant. Thermal/hydro plants stay locked together through physics because there's a prime mover connected to a generator, but solar plants rely on computer-managed simulations of the rotation that tries to follow the rest of the grid. When you have more simulation than reality it can get tricky fast.
Less neutrally, there have been far too many years of people pretending that you can build a grid out of 100% renewables. It was a false claim from the beginning, but engineering was overridden by ideology. The Iberian outage was one of the consequences - unfortunately merely a preview of coming attractions, as our political class is in denial and would rather believe lies than address the truth that it's caused by their own laws. Now it's only a matter of time until the entire UK or European synchronized grids experience a collapse of their own, with a similar root cause. We can't say we weren't warned.
For instance, a generator failure that cannot be mitigated by dropping load will probably result in islanding the grid: circuit breakers trip some lines and your network becomes several sub-networks. The sub-network that tripped can then be re-powered without a black start.
An important part is that clients pre-fail most requests and don't send them if the server is overloaded. They just assume the server would return an overload error.
can we blame it for the bug? can we credit it for the fix?
might be the most important part of the full report
Also this is karma because you blocked in ignorance entire datacenter networks making SMBs unable to deliver messages to your networks.
Ta-ta!
So to recap:
1. They did not have proper cell-based architecture to partition their system into failure domains ("In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure.").
2. They did not have proper monitoring to see the encroaching collapse.
3. Stuff just keeps falling off: "Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day"
Extent of my knowledge is nginx or traefik load balancing which at this scale is virtually equal to no knowledge. But seems like there would be systems in place to simply drop a sliding percentage of excessive traffic or block some regions once critical thresholds are reached rather than compounding the problem with forwarding the growing number of requests downstream to next victim server.
The easiest, and the default choice when I worked at Google, was exactly what you said: a sliding scale.
You could target racks, data centers, clusters, regions, etc. And “drain” the unit. For instance, stop routing traffic to cluster foo.
Then you could increment up stepwise 10%, 20%, etc.
Obviously we didn’t use nginx, and sure there’s interesting engineering involved. But it’s not like these systems are magic. There’s just a lot of clusters and a lot of load balancers. But at the end of the day, it’s just a load balancer.
As another Xoogler in this thread pointed out, there’s two problems:
1) Sometime teams accidentally introduce circular dependencies. Service foo might require service bar to boot and vice versa. Typically that’s fine, but from a rare cold start where both foo and bar are down, you’re in a pickle.
2) Load balancing is “easy”. But the services need to be designed with the aforementioned load balancing strategy in mind. Stuff gets tricky if there’s any pollers, process affinity, multiple components that need to process parts of the same user’s request in tandem. Given that outlook is ancient, I would not be shocked to learn that the system doesn’t respond well to naive “slowest ramp up traffic to a cluster” approaches. For big distributed applications, it takes a lot of meticulous design to build a system that responds well to simple SRE techniques like that.
That is one thing my team at Google would actually consider. We would make the code and architecture more complicated to accommodate the types of naive mitigations one would want to apply in an outage.
In 2014 Microsoft cancelled the dedicated SDET (Software Development Engineer in Test) role, triggering significant layoffs.
Before Linux took over the server market in 2000, Windows server was the dominant player in mid-market (enterprise market has IBM mainframes too). But you would have be old enough to remember that decades ago.
I use it since day 1
How would you say it has degraded?
Typescript paused new feature development to port to a new implementation language. It's too early to see if it's going to be better or worse for TS devs, will probably need to wait a few years (at least after the easiest things to pick are gone) to see if it was worth it for anyone who isn't building vscode sized projects.
VSCode was great, until LLMs hit the scene. Now there's a new update every 2/3 days either shoving more LLM-ware to the point that the seams are breaking, or it's a security bugfix for something that should never have been added in the first place.
You'd see occasional crash screens, but given the sheer number of machines out in the wild, having only a few crash once in a blue moon speaks amply to the reliability of the thing.
"The FDIC OIG is alerting banks about an increase in a type of fraud known as “ATM Jackpotting.” ATM Jackpotting involves criminal actors accessing ATMs and placing malware software on the system, enabling them to illegally dispense cash from the bank’s cash supply. The process is often run by sophisticated and organized crime rings at both the physical location (accessing machine and hard drive) and operating in foreign jurisdictions (transmitting malware codes).
The FBI estimates 700 instances of ATM Jackpotting and losses totaling more than $20 million in 2025."
https://www.fdicoig.gov/atm-jackpotting
https://therecord.media/fbi-atm-jackpotting-2025-report
An ATM is actually simpler than a fruit machine. The latter has more reasons to refuse a payout!
Unfortunately, as I understand it, it basically needed to pretend to be Win 9x on release due to Microsoft's (understandable and smart) focus on backwards compatibility, so there's effectively a layer on top of NT that hides its most powerful capabilities. Then there's all the cruft Windows layers on top of the kernel, so one could be forgiven for missing the gem sitting beneath the mound of manure.
https://blogsystem5.substack.com/p/windows-nt-vs-unix-design
From what I've read, Windows NT does have a good design underneath while Unix is simpler, but I don't think any windows version has offered the capabilities in a straightforward manner and easy to administrate. Too much awful layers as the parent comment says. While Unix has been mostly the kernel => the service manager => your service surrounded by a few concepts like users, unix sockets, file permissions,...
These technologies are battle-tested, and were adopted or replicated in Linux-land . Fore example, systemd does event logging and service management. Likewise, Linux file permissions haven't really stood the test of time, enterprise-grade systems use ACLs, which again, are a retrofit into Unix/Linux, but native to windows. For Linux, you have a choice of these implementations, but most are built on top of the OS, and each have their fanbases.
The 'traditional way' of sysadmin-style semi-custom Linux boxes, each with their own permissions setups and init scripts have thankfully gone out of vogue.
The creators of UNIX inherited the hierarchical file system of Multics, but they have abandoned ACLs, considering that they are too complicated for small mini-computers with a few users, and they have replaced the ACLs with the well known UNIX file permissions.
So ACLs are not some Windows or Microsoft innovation. Both the POSIX ACLs and the Windows NT ACLs have resurrected the Multics ACLs.
Regarding system logs and service scheduling, Windows has made only negligible improvements over UNIX.
Neither are really recognizable from their "roots" these days. The only certainty is that updating Windows still takes bloody ages.
Today I took an Ubuntu VM from 24.04 to 26.04 - it took about 10 minutes (that includes downloading the updates). It was a fairly simple system and I didn't have to repair anything. The service was actually up for most of the update.
I've run Exchange updates that have lasted most of an evening and night and that was on a single box Exchange on prem deployment.
On Gentoo Linux an emerge -Uva --deep --newuse @world can take a little longer than a Windows update sess but you are compiling the OS from source 8)
DEC even sued Microsoft and Microsoft settled (sound familiar? DOS? Can't Microsoft invent anything of their own?).
Hey, they got it quite reliable eventually (Windows 2000 timeframe), but still.. To say Linux is 1970s tech and WinNT ISN'T seems a real stretch.
And if other more modern kernels offer significant improvements over them, I wonder why we are not seeing them. Should be easy enough to add a Linux or Windows syscall layer to a modern kernel like that.
In the beginning, the NT kernel API was a mixture a functions taken from VMS (mostly for asynchronous I/O and thread/process management) with functions taken from UNIX (mostly for classic I/O), where most functions had various corrections or enhancements in their lists of parameters in comparison with their models from VMS and UNIX.
The later Windows NT versions added some innovations, e.g. I/O completion ports in Windows NT 3.5, waitable timers in Windows 4.0, and so on, but there were not many such innovations and most were not very important.
Around 2000, one could claim that Windows NT is a much more modern OS than Linux.
However, after 2000 Linux started to add a lot of innovations over what is specified by POSIX. Today, any high-performance application running on Linux that uses the POSIX API instead of using Linux-specific features, makes a big mistake.
Nowadays, for many applications Linux has a better API than Windows, but the Microsoft developers are not stupid, so in recent years they began to incorporate into the Windows kernel some of the more important Linux kernel innovations, like futexes and io_uring.
Somehow it manages to be slower in uncontended cases so maybe in hindsight it wasn't that good. But on Linux you get permanently stuck threads if your NFS server is down.
Consider: exclusive file locking for almost everything. UNIX lets you delete or move an open file, the Windows kernel doesn't and this creates enormous complexity at higher levels of the userspace stack especially for any kind of auto updater. It's also one reason Windows updates are so painful.
Windows has pathetically low performance in the filesystem layer, largely thanks to its filter driver design. It's not uncommon to find apps written originally for Linux are basically un-portable to Windows because they assume fast file IO and Windows just can't provide it. This was also a major motivator of the WSL2 effort. Windows actually has a feature called Dev Drive. It's a whole alternative filesystem that's faster than NTFS. They call it Dev Drive because they assume only devs need good filesystem performance (gah). You can't run Windows itself on a Dev Drive though, because of tech debt-y "reasons".
This ridiculous approach of assuming developers are special is carried over to other things too. Windows supports symlinks but this support is nearly useless, because it's disabled out of the box for security reasons (symlinks play a role in some kinds of attack on Linux). So you have to turn on Developer Mode in the control panel to be able to use them.
Sometimes you'll see the Windows approach to security be praised (SIDs, object manager etc). It does have a more principled design than Linux, that's true, but the standard here is macOS/XNU not Linux and the Windows kernel's sandboxing support is far worse than macOS.
Windows processes are extraordinarily heavy compared to a Linux process. They take more RAM, start up much slower, and context switching is slower. Again, this difference often renders Linux software un-portable because they assume they can fork and exit processes very fast, but on Windows you just can't. You pretty much have to replace a multi-process design with threads.
Despite the Windows kernel not being a competitive advantage to Microsoft (IMHO), and despite that they run public symbol servers, they keep it closed source. If you really want to understand the Windows kernel you can do so down to the level of individual function calls with public information, but you can't actually read them. The reason is probably security-through-obscurity for DRM, Warbird and other such things, not that it seems to help much. This makes it frustrating to figure out what's going on.
Relatedly, the Windows kernel is an absolute war zone. It's nothing like Linux, which is a conventional textbook OS design. On Windows the kernel is riddled with deliberate traps, code obfuscation engines and "tilt sensors" designed to try and block attempts to hook internal functions by malware or other kinds of malicious software. This shows up in other places too: for example, the API to Linux is the system call instruction. On Windows the kernel theoretically doesn't have a stable syscall interface and you're supposed to go via public APIs like kernel32.dll, which in turn just forwards to ntdll. In practice it does have stable syscalls because MS never randomized the system call interface, so apps trying to detect hooking in userspace do invoke system calls directly (often for anti-cheat purposes). The result is there are like three different ways to invoke NT system calls, all of which are extremely overlapping and duplicative (kernel32, ntdll, system calls), all of which are used by apps in reality but only one of which is documented.
- File locking is a convention on Windows, not a necessity and you don't pay for it if you don't lock files (which is an unfortunate default btw).
- By filter drivers being slow, I think you mean antivirus and security stuff, which are an unfortunate reality of daily windows use, but aren't really caused by filter drivers being a thing. If you needed antivirus on Linux, I think you might incur a similar penalty
- I think symlinks would be nice, but I haven't ever really missed them on Windows. The things linux uses it for, like linking libraries are done differently on Windows
- Devs are special in some sense. You can't afford to break existing workflows on users machines. Just recently, the EU decided to replace Windows with Linux, discovered that stuff doesn't work, then promptly went back
- I appreciate that you call out Windows security as a positive, but as with many other subsystems, Windows has shipped with many systems that were production grade and built in, like security, ACLs, event logs, etc. etc. which are not part of Linux by default, and are the subject of endless debates as to how to implement them (some of them, like systemd, mirror Windows functionality)
- A concept of a Windows process doesn't really map to a Linux process. Linux processes are isolated threads, while Windows ones are containers (and don't even necessarily have a main thread). And afaik Windows processes aren't that heavy any more. Their heaviness came from loading gdi32 and user32 dll, as they expected to host GUI apps, on new Win versions, that's no longer necessary. But fork() style process management is foreign to Windows, and the one place where its useful in Linux, cli apps, Powershell has an entirely different design, where its more like a JITed scripting language, and ls or cp are not separate processes.
- I don't think heavily multiprocess designs are in vogue anyways (at least not how Linux does them), but in terms of RPC, COM multi-process RPC was much more efficient in Windows compared to things like eventfd. Android even added binder, which is a similar mechanism
Many of Windows' mechanisms are a bit long in the tooth, but arguing that Linux is better, then adopting a very similar mechanism in Linux years to decades later doesn't really show that engineering superiority.
I'm not a Linux hater btw, except in one thing - the fact that there's no good standard way to build and distribute binaries that can be expected to work on any reasonable Linux install is criminal.
IIRC it's more that the filter architecture itself is slow, but yeah Windows comes with some slow filters out of the box that don't help. There's a writeup somewhere about this from a Windows kernel engineer but it was years since I read it. AV is indeed the worst offender but Windows always has one, so <shrug>.
systemd is inspired by macOS Launch Services. I guess svchost is the closest equivalent in Windows, but it didn't play any role in the systemd design process.
Linux processes do contain threads, I didn't quite understand what you meant by saying they're not containers. There is now the concept of a Windows picoprocess but it can't do much and isn't used after WSL1 was removed, if I understand correctly. I agree that Windows doesn't really need cheap processes architecturally and native apps don't rely on them without that imposing much cost.
[D]COM isn't a kernel feature but the closest equivalent on Linux would be DBUS, not eventfd. I don't know how they compare in terms of efficiency, I'd expect about the same, although in a past life I actually worked on an open source implementation of DCOM!
I didn't argue Linux is better, I have plenty of beefs with how Linux works too :) The Windows kernel is rightly very popular because it offers a stable driver API, which in turn enabled a huge gaming and hardware ecosystem, for instance. But architecturally I don't see big differences between them tbh.
Now Midori, on the other hand... that really was different!
I see that in general Windows takes a "big monolith" approach to things that Linux can handle as a large array of small pieces. An app can be a few files on Windows, a help database is one file, a source code repository would be one file if all the VCSes weren't imports from Linux.
This might be Conway's law in action. On Windows you were more likely a proprietary shop writing software that just does what it should do - if you needed to change it then you changed the software. On Linux because it's a bazaar you have a lot more of trying to tinker with the things that are already there, or add on small pieces. The average unit of deployment on Microsoft platforms is (or was) Turbo Pascal or Visual C++ 6 or QuickBooks or Microsoft Office; the unit of deployment on open-source platforms is grep or sed.
But that's MY perception of that period of time. I intentionally avoided working for Windows-based places because it sucked so bad.
I agree. We're talking web servers here, not SMB file and print servers, which were mostly WindowsNT based and the contrasting vendor would be Novell until sometime in the late 90s.
The vendor of popular web server software over time is well documented thanks to 30 years of Netcraft. Unfortunately, we don't see the platform, but most uses of Apache were either Linux or Sun based, and Microsoft shops tended to use IIS and rarely anything open source. My own experience at the time was that things that ran on Sun was mostly java based, WebLogic.
https://www.netcraft.com/blog/march-2026-web-server-survey
According to the graphs:
Microsoft's share of the top 1 million busiest sites was never above 20% and has been steadily declining for the last 15 years. If you were serving a lot of traffic, you weren't (and aren't) using Windows to do it. And the majority of traffic has always been served by something other than Microsoft.
Microsoft reached over 50% in 2017 in marketshare of all sites, but traditionally has been limited to below 40% and has really dropped off since 2020. And (based on the graphs), that traffic probably migrated to nginx.
There was a period in here where Netcraft had to make a distinction between sites and servers. Cheapo web hosting services via virtual hosting was on the rise, and a massive number of domains/sites were all served from the same server, and it skewed the numbers in terms of vendor market penetration (think shared-hosting offerings like GoDaddy, the hosting industry was lousy with them). The back and forth churn was notable when some big hosting company switched their web server software and then hundreds of thousands of sites changed software vendors. I remember Microsoft making a big deal about this with marketing copy like "The most popular web server software on the Internet", which may have been true based on domains/sites, but not based on installed instances or most-served-traffic.
IME Microsoft was never a big vendor of web server software, unlike how they are a big vendor of desktop. People were aware of IIS as an option if you were a Microsoft shop, and I know some people who ran it (I once crashed my buddy's site using the perl WinNuke one-liner to put him in his place after bragging about his NT and IIS setup -- hi Noel!) but Sun dominated and where it didn't, it was open source solutions.
> I intentionally avoided working for Windows-based places because it sucked so bad.
Same.
The problem is making all the moving parts from all those people work together well. Weakest link is almost never the people but the complexity of everything.
Does that mean Microsoft's internal code are 100% outsourced to agents / AI and human review just won't be able to catch latent bugs like this?
Looks at my Windows Task Manager...
For example just accept connections from specific (random) IP ranges. They could even cycle through different ranges every 5 minutes to give clients a chance to sync at least once every hour.
The reduced load could be used to gradually bring up servers again without failing, add additional capacity, and gradually restoring access. Completely without fixing the bug at all. The system managed to handle usual load even with the bug, it just hit some critical point that made it collapse. Adding more resources would also fix that.
Restricting external access wouldn't solve this issue.
I guess adding a config option for throttling queue processing is something any distributed system should have.
Most big cloud services operate several layers of load balancing. The first is a smart DNS server that turns a lookup of mail.outlook.com or whatever into an IP address of a datacenter roughly nearby to where the user is, to a datacenter that's not too overloaded. But because DNS lookups are cached and sometimes cached too long by clients this is a very rough and ready form of LB.
The second layer are routers that spread requests to a single IP address over "hardware" LBs that take raw TCP/IP connections from the internet. Then those machines do some basic health checking and forwarding to the frontend web servers. At that level the machines aren't aware of what service or customer the connections are for, they're just doing some flow hashing to pick a "backend" (another frontend).
Then the cloud/service level machines terminate the TCP/IP connection and often TLS as well. They're reverse proxies that do URL matching and forward traffic on to the app specific servers. This is the first layer at which you can start to intelligently inspect or drop traffic.
No one person in a big org has the ability to limit traffic to a service across every layer of load balancing that exists. If the reverse proxies start dying, reducing traffic would mean changing the configuration of the hardware LBs which is usually a different team.
I'm guessing there must be some deeper issue at play making it too risky or not possible. Maybe the hardware LB's were too intertwined with customer Azure loads or something?
But when you are hitting hour 12 of a global outage of one of your most central services I would assume even quite exotic solutions would be considered (e.g standing up dedicated hardware LB's with load shedding and pointing some of the DNS resolution to those)
This looks like a potential weak point against a cyber attack. A big number of distributed bots in one geographical area can send their requests to one data center and overload it.
Does anyone know why the situation got worse on better machines instead of worse? Normally it is the other way around when it comes to test coverage of hardware specs
I don't think anyone would be astonished by this bug - lock contention is hard. https://www.itnews.com.au/news/kernel-crashing-noisytcp-bug-... is a recent example in the Linux kernel.
This is one of the areas that provably correct software can make a big different and is worth the cost (even before AI)
That whole series reads like sour grapes by someone bitter about how they left.
Perhaps Microsoft will learn from this now they see the direct financial cost themselves of sloppy code.
I will still get sent to junk 70% of the time, especially to Microsoft Exchange
I'm an IT admin at my company (though I don't often actually admin the MS parts), so I regularly receive e-mails from [email protected] that almost always end up in the junk folder. The mail is legitimate. But it's typically about such-and-such doing something that's against some policy, so we get notifications. Since I'm not in charge of either side of the policy violation, I could never be bothered to look into why these messages end up in the junk folder.
Why would AI be at fault ? We are talking about Microsoft here.
Throwaway account, I work at Microsoft, everything was AI's fault.
I think the deafening silence of "nobody asking" the question speaks for itself.
Seeing AI "fix" the code makes everyone just turn off the brains and move on.
Those here who program know - no one reads code or PRS.
Seeing AI "fix" the code makes everyone just turn off the brains and move on.
Those here who program know - no one reads code or PRs.
We saw with our own tools back when an artist were given a low-core "server" like machine with many CPUs (good we thought), and 2 NUMA nodes, and suddely the level editor for our game was running much slower. We could blame std::shared_ptr<> all day long, or similar but it's there and it was real.
Tried the 4kb interleaved paging, but then everything was equally slow. At the end we simply blocked the App to run only on one NUMA node (still using memory from the other one), and things were back to "normal". "Normal", because promises were not kept - we could not use all CPUs.
Stuff like this also happened prior to AI. One of reasons it happens more frequently now is just now timelines of product development are compressed. So let's say we get 3x outages, but at 10x the speed of new features.
To me it sounds like Outlook team do not have proper gradual rollout for service of this size. Like rollout the new version to 10% servers, let it soak, compare metrics before rolling up higher. Most of "bigtechs" have something like this nowadays.