News for hackers. Stuff that matters.
Re:Microsoft blames monster email outage on software defect ... by mike_hearn on Monday November 16, 2026 @10:42AM (#50518809)
No big surprises here. Big outages at big firms seem to always look like this.

The electricity industry has this notion they call a black start, where they bring the grid online from zero. It's a very difficult operation because you can't start a generator straight into an offline grid otherwise it gets hit with full demand and can't keep frequency in the right range, which can damage equipment. So they have to start by disconnecting all the loads and then bringing load blocks back in sequence, trying to roughly balance available generation with the size of the next block. In practice black starts are extremely rare, but they can happen after a cascading failure in which something causes some generating stations to trip, redirecting load onto the remaining stations which can't handle it causing them to trip in turn. The entire grid can collapse in just 60 seconds or so once that kind of runaway failure starts.

This kind of outage is very analogous to a black start. A few servers get into trouble - maybe they're in a crash loop or maybe their caches are cold - the load balancers shift traffic onto the remainder which then can't handle the load either and so the failure cascades. Once a service is in this state sometimes the only remedy is to disconnect all load, bring the servers back to stability, then bring load blocks back online in small chunks, making sure to never hit anything with the full load immediately to ensure any caches or JIT compilers have time to warm up.

Black starting a service is obviously hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster. I used to be an SRE at Google and although we regularly started up new clusters and moved traffic around between them, actually starting an isolated service instance entirely from scratch was never something we practiced. I remember in a few cases teams discovered devs had introduced circular dependencies by accident, i.e. starting service X required service X to already exist, or it depended on service Y that in turn depended on X. Sometimes you can only find these things by practicing.

Re:Microsoft blames monster email outage on software defect ... by EGreg on Monday November 16, 2026 @11:52AM (#50519724)
In software servers this is called the “thundering herd” problem. It is why clients are coded do not just exponential backoff but random offsets too.
Re:Microsoft blames monster email outage on software defect ... by chinathrow on Monday November 16, 2026 @12:22PM (#50520100)
It's also a well known problem, so no excuse for that.
Re:Microsoft blames monster email outage on software defect ... by bravetraveler on Monday November 16, 2026 @03:51PM (#50523075)
Business continuity/disaster recovery should also be well-known/defined, agreed. No excuses.

I did SRE at another firm, not Google like the parent but a similar level. We tested these procedures twice a year and every time we added a region.

Re:Microsoft blames monster email outage on software defect ... by dijit on Monday November 16, 2026 @04:38PM (#50523675)
a well known problem with more than a few known fixes, one of which is load shedding your upstream based on protocol aware load balancrs.

I agree: no excuse.

Re:Microsoft blames monster email outage on software defect ... by VirusNewbie on Monday November 16, 2026 @05:49PM (#50524614)
Not only did they not handle thundering herd, it happened across all regions.
Re:Microsoft blames monster email outage on software defect ... by olalonde on Monday November 16, 2026 @05:52PM (#50524656)
Also, circuit breaker pattern.
Re:Microsoft blames monster email outage on software defect ... by rawgabbit on Monday November 16, 2026 @03:29PM (#50522779)
[delayed]
Re:Microsoft blames monster email outage on software defect ... by solyin on Monday November 16, 2026 @05:37PM (#50524464)
Did you do anything to try and fix that?
Re:Microsoft blames monster email outage on software defect ... by rawgabbit on Monday November 16, 2026 @06:31PM (#50525108)
[delayed]
Re:Microsoft blames monster email outage on software defect ... by ryanburk on Monday November 16, 2026 @08:30PM (#50526236)
I can't speak for current MS cloud services, but we absolutely used to do this in the hotmail/outlook days. we had the notion of a "rack" of services that combined to compose the service plus a number of accounts, and before an update we would remove the scaling so it became more heavily loaded with real traffic patterns and see if it fit the current expectations or anything changed.
Re:Microsoft blames monster email outage on software defect ... by imron on Monday November 16, 2026 @11:37PM (#50527506)
LGTM
Re:Microsoft blames monster email outage on software defect ... by xunil2ycom on Monday November 16, 2026 @03:44PM (#50522988)
Well stated.
Re:Microsoft blames monster email outage on software defect ... by jvidalv on Monday November 16, 2026 @04:00PM (#50523208)
Happened to Spain last year. The black start.
Re:Microsoft blames monster email outage on software defect ... by tonfa on Monday November 16, 2026 @04:18PM (#50523434)
Somewhat related: https://company.sbb.ch/en/railway-development/services-rus/e...

SBB just finished their project to be able to restart the grid in 5min. (There was a famous multi hour outage a few decade ago that I think motivated this work)

Re:Microsoft blames monster email outage on software defect ... by nickdothutton on Monday November 16, 2026 @05:01PM (#50523996)
3-5 days in the UK according to "regulations". Some talk of it potentially taking 7-14 days under certain bad circumstances.
Re:Microsoft blames monster email outage on software defect ... by mike_hearn on Tuesday November 17, 2026 @04:40AM (#50529344)
I think 24 hours is more likely for the UK unless the outage was caused by a major storm that damaged equipment, or sabotage. A few isolated areas might take a few days to come back.

If the outage involves damage to transformers then all bets are off. In those cases some areas of the country might become uninhabitable for months.

Re:Microsoft blames monster email outage on software defect ... by dubcanada on Monday November 16, 2026 @04:29PM (#50523555)
That's not a 100% how this works, a load balancer setup should take into account a "server being ready to accept traffic" that includes cache and JIT.

A power grid is different because it's not about the power immediately being used all at once, it's about stabilizing the frequency. Turning generators on takes a while to stabilize, each reactor may output inconsistently for a while, which is why you never start a generator and immediately ask for full output, you start it, let it warm up, then add load. Which does sound like a server, but you don't need every server speaking the same frequency.

If a grids frequency is off, or stutters it kills electronics. If a servers JIT cache is ready, it's slow. The extremes are completely different.

Now the same garden hose problem exists, going from 0 to everyone logging in at once is a lot of load. But that's why you come back online with more servers and scale back.

Re:Microsoft blames monster email outage on software defect ... by brunoborges on Monday November 16, 2026 @04:54PM (#50523884)
The biggest challenge of bringing a resilient system back online after an outage, is the resiliency of said system fighting back.
Re:Microsoft blames monster email outage on software defect ... by Rygian on Monday November 16, 2026 @05:02PM (#50524006)
Back in 2025, Spain did a very-dark-brown-almost-black start (with help from neighbours), starting with one hydropower plant and increasing from there.

It went remarkably well.

Re:Microsoft blames monster email outage on software defect ... by tumetab1 on Tuesday November 17, 2026 @03:50AM (#50529038)
In Portugal it didn't go that well, as expected since real situations differ from tests.

The initial start up (of the cold start plants) took the double of time expected. The "cold start" capacity is being increased due to some of the issues found.

Re:Microsoft blames monster email outage on software defect ... by ncruces on Tuesday November 17, 2026 @04:22AM (#50529239)
It actually was better than in Spain. Large portions of Spain were left without power for almost 24h, despite the help from France (initially) and Morocco (later).

Portugal was able to recover in 12h (still a lot) without any external assistance.

Re:Microsoft blames monster email outage on software defect ... by Rygian on Tuesday November 17, 2026 @07:14AM (#50530376)
> About 16:00 CEST on 28 April, Spanish electrical operator Red Eléctrica de España (REE) estimated it would take "between six and ten hours" to restore service

> By 07:00 CEST (05:00 UTC) on 29 April, electricity had been restored to 99% of energy demand, and full restoration was achieved by 11:00 CEST (09:00 UTC) that same day.

The wikipedia references seem valid for this precise timeline, which is more informative than your remark.

https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou...

Re:Microsoft blames monster email outage on software defect ... by ncruces on Tuesday November 17, 2026 @08:55AM (#50531415)
Power was fully restored in Portugal by 22:22 UTC on April 28: https://eepublicdownloads.blob.core.windows.net/public-cdn-c...

I'm not trying to win a pissing match, just trying to counter the very common miserabilist notion that Portugal did worse than Spain.

This specific problem was (1) caused in Spain, (2) fixed faster in Portugal, and (3) without external help.

Portugal could've been the cause, and could've done better, for sure.

But it's just not balanced to write "In Portugal it didn't go that well" as a response to "It went remarkably well [in Spain]." Makes it sound much worse than it actually was.

Re:Microsoft blames monster email outage on software defect ... by fransje26 on Tuesday November 17, 2026 @05:38AM (#50529725)
It's the part before the restart that didn't go well at all...

Do you have any engineering insight on why a hydropower plant was used, and not, say, a gas-fired power plant?

Re:Microsoft blames monster email outage on software defect ... by mike_hearn on Tuesday November 17, 2026 @10:16AM (#50532510)
Power stations need power to start up and operate, called house load. It's for things like pumps and excitation current. To black start a station requires the house load to come from somewhere, like a generator set. For a coal, gas or nuclear plant the house load is quite large. For a hydro plant it's small, you just need enough to open the wicket gates and energize the generator coils. Therefore, black start capability is cheaper to sustain at hydro stations than other kinds.

Once the hydro station is back online the power it generates can be used to provide house load to the other stations.

Re:Microsoft blames monster email outage on software defect ... by nialv7 on Monday November 16, 2026 @05:10PM (#50524109)
that's not the problem with grid black start at all? black start is difficult because many types of generators themselves need power to start. it's a circular dependency problem, not a load management one.
Re:Microsoft blames monster email outage on software defect ... by avianlyric on Monday November 16, 2026 @05:23PM (#50524277)
It’s both, but every grid has a sub-set of power stations with black-start capability (a crap ton of diesel generators), that exists to break the circular dependency problem and bootstrap the rest of the grid.

But load management while bootstrapping the rest of the grid is still very hard. Often the reason for really long blackout, ones measured in days rather than hours, is repeated cascading failures as operators try to reconnect parts of a the grid, and accidentally overload recovered parts of the grid, causing generators to trip off again.

Re:Microsoft blames monster email outage on software defect ... by hwillis on Monday November 16, 2026 @07:34PM (#50525766)
> every grid has a sub-set of power stations with black-start capability

Typically <5% afaik

Re:Microsoft blames monster email outage on software defect ... by tacticus on Monday November 16, 2026 @07:40PM (#50525816)
> It’s both, but every grid has a sub-set of power stations with black-start capability

They should have that and they typically pay substantially more for it. Unfortunately it's also the case that the private companies providing said black-start capability may have just not bothered to tell anyone they haven't been able to do it for a few months\years.

Re:Microsoft blames monster email outage on software defect ... by avianlyric on Wednesday November 18, 2026 @07:48AM (#50545682)
Nor should. They absolutely must, otherwise it would be basically impossible to blackstart the grid.

A properly run network will require those facilities to be tested multiple times a year to ensure it actually works, and make payment for the facility contingent on passing those tests.

Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @04:20AM (#50529223)
Wikipedia says black start stations are often hydro dams, since all they have to do is open the floodgates (literally), without any preheating or fuel pumps or anything like that.
Re:Microsoft blames monster email outage on software defect ... by bell-cot on Tuesday November 17, 2026 @10:09AM (#50532427)
Hydro is ideal - if you have a good-size hydro plant handy, where a sustained drought can't possibly drop the reservoir too low for the penstocks to function properly.
Re:Microsoft blames monster email outage on software defect ... by avianlyric on Wednesday November 18, 2026 @07:47AM (#50545665)
Can’t be any old hydro plant however. Big hydro plants will be using turbines is electrically excited magnets (I.e. electromagnets), but that’s only way to build very high power turbines without tons of expensive and fragile permanent magnets.

So you need a hydro plant that is either small enough that it’s using turbines with at least some permanent magnets (you get turbine varieties that basically have a mini turbine inside using magnets to power the energise the larger electromagnets), or has dedicated infra to produce power to energise the coils in their turbines before they’re producing enough power to energise themselves.

Re:Microsoft blames monster email outage on software defect ... by bragh on Monday November 16, 2026 @05:34PM (#50524431)
That was a very enlightening moment in Factorio when I had already converted everything to power and the whole grid blacked out and I needed to get it going somehow again.
Re:Microsoft blames monster email outage on software defect ... by hwillis on Monday November 16, 2026 @07:33PM (#50525757)
It's certainly a problem with black starts: https://fdehydro.com/the-black-start-blueprint-how-power-gri...

The process is called islanding.

Re:Microsoft blames monster email outage on software defect ... by scheme271 on Tuesday November 17, 2026 @12:55AM (#50527935)
The other issue is that you need to be able to sync up the generators. E.g. your power plants are running at 50Hz or 60Hz or whatever. But are they running in phase, if not, then how do you make sure that each new generator is in phase with the overall grid in order to avoid the generators from fighting each other?
Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @04:21AM (#50529232)
For that they use a couple of analog knobs to adjust the generator speed, and a specialized type of analog meter called a synchroscope, which measures phase difference. When it's at zero and not moving, they pull the big lever.

This is very much like trying to jump onto a train from another train moving alongside it.

Re:Microsoft blames monster email outage on software defect ... by mike_hearn on Tuesday November 17, 2026 @04:52AM (#50529421)
Maybe in the 1920s ;) Pretty much any power station built post-war will have automatic synchronization. You aren't going to manually synchronize a large power plant these days, it's far too dangerous. Even very old plants have safeties that prevent closing the breakers if the sync isn't close enough.
Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @06:01AM (#50529866)
Makes sense, I saw it on a YouTube video of a relatively small station.
Re:Microsoft blames monster email outage on software defect ... by CobaltFire on Monday November 16, 2026 @05:10PM (#50524116)
I never worked in civilian power generation, but I can vouch for this on Nuclear Ships.

We had to practice load pickup on a normal start at the pier often, but we likely practiced a full SCRAM to back online (including a FULL load shutdown) more.

Having predefined lists of what gets brought up at various points helps a ton, but it's still not an easy or simple operation.

Re:Microsoft blames monster email outage on software defect ... by TomatoCo on Monday November 16, 2026 @06:29PM (#50525088)
I imagine the predefined list is easier when it's all consumers that are under your control. I figure with civilian power some random percent of houses'll kick their AC on after power is restored.
Re:Microsoft blames monster email outage on software defect ... by noir_lord on Monday November 16, 2026 @09:13PM (#50526536)
It's why they bring up a subset a bit at a time and fewer houses/businesses than would be expected under average load, Practical Engineering has a fantastic video on black starts[1]

[1] https://www.youtube.com/watch?v=uOSnQM1Zu4w

Re:Microsoft blames monster email outage on software defect ... by lifeformed on Monday November 16, 2026 @09:58PM (#50526837)
I never worked on nuclear ships, but I can vouch for this in Satisfactory.

You need a temporary source of power to start the water pumps and it takes a while for the water to flow all the way through the system and for the coal to saturate all the belts and power plants, if you hook it up to the main grid before all the plants are operational then it shuts down again.

Re:Microsoft blames monster email outage on software defect ... by epiccoleman on Monday November 16, 2026 @10:35PM (#50527095)
heh. this is why I've got programmable speakers hooked up to the belt that feeds fuel to the steam generators in my Factorio world - because I had one too many times where I moved something or messed up my logistics and had to cold start the base by hand feeding coal to get back online.

Factorio makes this fairly easy for the most part, a cold start isn't really some massive engineering challenge, but still, worth guarding against. By the time you get solar and / or nuclear power is basically solved, at least on Nauvis, but it's "overengineering the game," so... we overengineer.

Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @04:14AM (#50529187)
FYI you can put your coal mining stuff on a separate power network, and then an overload on the main network won't cascade to it.
Re:Microsoft blames monster email outage on software defect ... by pjc50 on Tuesday November 17, 2026 @04:53AM (#50529426)
Yeah, Gleba seems deliberately set up to make brownouts risky (letting your supply of trapped pentapod eggs hatch and eat your base) and black starts harder - base fills up with spoilage which needs to be removed. At least you can burn the waste for fuel.

In Factorio, solar is an easy source of cold start. In the real world, I think hydropower is a commonly designated choice and the solar farms have to wait for a grid to sync against.

Re:Microsoft blames monster email outage on software defect ... by marcosdumay on Tuesday November 17, 2026 @12:10PM (#50534587)
Power switches are your friend.

Cold starts are a constant in Aquilo until you research and build fusion power. And whatever you do by hand to handle them, there's a device in the game that does it automatically too.

Re:Microsoft blames monster email outage on software defect ... by epiccoleman on Tuesday November 17, 2026 @02:55PM (#50537204)
Neat, I haven't gotten to Aquilo yet, but I understand it's quite a challenge. Currently working through vulcanus in my first space age save.
Re:Microsoft blames monster email outage on software defect ... by Beretta_Vexee on Tuesday November 17, 2026 @04:50AM (#50529400)
It’s much the same principle as with an electricity grid, but on a different scale. We start up diesel generators to start gas turbines, which in turn supply power back to the distribution substations.

Once we’ve restored power to the substations, we can carry out what are known as voltage restoration operations.

At a nuclear power station, if all goes well, there will be at least one nuclear reactor unit that has managed to operate in island mode. They have their own sub station and can restart the others units.

We restart as many units as possible on a virtually empty section of the grid (serving only essential and vital consumers such as hospitals). We then reconnect consumers in a predefined order to avoid significant fluctuations that could trigger further cascading failures.

The most difficult part is often identifying the exact cause of the blackout with certainty, so as to prevent it from repeating it.

Re:Microsoft blames monster email outage on software defect ... by 2muchcoffeeman on Monday November 16, 2026 @05:28PM (#50524347)
Good to know even Google introduces circular dependencies!
Re:Microsoft blames monster email outage on software defect ... by nilptr on Monday November 16, 2026 @05:37PM (#50524461)
There's literally no such thing as "black start" at Microsoft that I am aware of. The way services are brought online initially is through a build out process coordinated through hundreds of Teams. It's an organic, messy, and painful way of doing things. As such, build outs are a human driven process with bits of automation spanning sometimes 20+ years of engineering touches, troubleshooting guide, and some additional tribal knowledge thrown in passed down through the years. There is little consistency across teams and sometimes even within a team as every Azure Cloud is effectively slightly "different".
Re:Microsoft blames monster email outage on software defect ... by markus_zhang on Monday November 16, 2026 @10:43PM (#50527158)
Reading this, I can literally hear people shouting over the mic.

“Ready? 10 seconds to power on server alpha-romero-one-o-four!”

“Load—at five percent, check! All systems stable!”

Re:Microsoft blames monster email outage on software defect ... by nilptr on Monday November 16, 2026 @11:18PM (#50527379)
Ha, no. It's not quite like that, but it can take weeks to bring up a service especially if you're in the lower rings. Often time core cloud stuff is being built out while services are also building out so stuff gets stubbed out and then you have to go back and slowly fill in the dependencies.

It's chaotic and stressful in a slow asynchronous way.

Re:Microsoft blames monster email outage on software defect ... by markus_zhang on Tuesday November 17, 2026 @08:45AM (#50531283)
I seem to get some morbid pleasure out of it.
Re:Microsoft blames monster email outage on software defect ... by bitwize on Tuesday November 17, 2026 @11:26AM (#50533782)
"Type 'cookie', you idiot. I'll head 'em off at the pass."

"Rabbit, flu shot, someone talk to me!"

Re:Microsoft blames monster email outage on software defect ... by coliveira on Monday November 16, 2026 @11:50PM (#50527600)
Nobody should ever depend on things done by Microsoft. It's in their culture to create things this way, it is totally undependable. My guess is that we still haven't seen the complete mess that MS can create with their online services, and everyone will pay for this reliance.
Re:Microsoft blames monster email outage on software defect ... by SV_BubbleTime on Tuesday November 17, 2026 @12:51AM (#50527908)
Sounds like the should reduce their domestic workforce and run more AI and H1B.
Re:Microsoft blames monster email outage on software defect ... by hobo123 on Tuesday November 17, 2026 @01:33AM (#50528146)
Something similar to Brooks' law - adding more H1bs to a project makes it later?

(I have no experience with H1Bs nor beef, I'm a German who's merely had the joy of seeing failed offshore projects.)

Re:Microsoft blames monster email outage on software defect ... by psd1 on Tuesday November 17, 2026 @05:28AM (#50529664)
Sure, but also, leftpad.
Re:Microsoft blames monster email outage on software defect ... by gosub100 on Tuesday November 17, 2026 @07:12AM (#50530362)
It's so bizarre they have no consideration for UX, or reliability or quality. But they will keep 16-bit APIs for backwards compatibility to DOS because some peanut butter factory in Indiana still uses it.
Re:Microsoft blames monster email outage on software defect ... by trollbridge on Tuesday November 17, 2026 @08:46AM (#50531313)
I’m not aware of Windows having any 16-bit DOS APIs since 32-bit Windows 10 was retired.
Re:Microsoft blames monster email outage on software defect ... by calvinmorrison on Tuesday November 17, 2026 @12:06PM (#50534519)
I'm in the middle of upgrading software released right before the invasion of Iraq. Things chugging along great
Re:Microsoft blames monster email outage on software defect ... by water42 on Tuesday November 17, 2026 @12:06AM (#50527678)
you're telling microsoft doesn't think ahead about having to cold start all services within a product during a worst case scenario outage?
Re:Microsoft blames monster email outage on software defect ... by jacquesm on Tuesday November 17, 2026 @12:29PM (#50534953)
I'm sure none of those people were amongst the 60,000 or so targeted in the recent round of employee reductions.
Re:Microsoft blames monster email outage on software defect ... by anankaie on Tuesday November 17, 2026 @01:59PM (#50536440)
There is a BCDR strategy for each service at different tiers (based on which "ring [of criticality]" your service is in). Every service is supposed to pass through a (yearly? biyearly? don't remember) compliance certification. Not sure which rings require practicing it, though.

All of it is documented through a unified catalog of services and their associated metadata.

Source: Worked in MSR and shipped services through Azure / CoreAI.

Re:Microsoft blames monster email outage on software defect ... by scheme271 on Monday November 16, 2026 @05:45PM (#50524552)
Texas was on the verge of having to do a black start a few years ago when winter weather started causing failures. I think people were quoting potentially weeks to get everything back online. As it was, I think people didn't have power for days during sub-freezing temps due to load shedding.
Re:Microsoft blames monster email outage on software defect ... by tcskeptic on Monday November 16, 2026 @09:55PM (#50526807)
Yep. I was without power for 54 hours. Weirdly, just across the street from me never lost power because they were on a priority segment with a hospital.
Re:Microsoft blames monster email outage on software defect ... by stephenhuey on Tuesday November 17, 2026 @12:44AM (#50527876)
After winter storm Uri, my house was without power for 4-5 days (we had gone to a friend’s house which was in a small oasis of light in a dark neighborhood because it was on a high priority bit). Long before, I was in a house a little over a mile from downtown Houston during Hurricane Ike, and spent 7 days in darkness and heat and humidity while gazing at the bright skyscrapers a half hour walk from my pitch black street.
Re:Microsoft blames monster email outage on software defect ... by toast0 on Monday November 16, 2026 @06:58PM (#50525389)
> Black starting a service is obviously quite hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster.

I mean... you can try, but unless you spend a lot of resources on it, I don't expect test load to be close enough to really practice a black start.

From experience, the best way to prepare for restoration from nasty outages is to have them frequently! The team I was on was pretty good at outage restoration, until we started regularly hitting about a year between outages. After a few years of one or maybe two outages per year, we had lost the muscle memory for protecting services that get overloaded during or after an outage and setting outage messages (which both informs customers and protects customer service queues).

Regularly starting up an isolated cluster at least lets you get ahead of sequencing issues though; hopefully it's enough if you can get the whole cluster started and then modulate inbound traffic (most complex services will need something like you mentioned so that you limit the number of sessions, but sessions that start are likely to proceed successfully).

Re:Microsoft blames monster email outage on software defect ... by hwillis on Monday November 16, 2026 @07:53PM (#50525935)
> and can't keep frequency in the right range, which can damage equipment.

This is one of those beautiful engineer understatements. Rotational speed is fundamentally linked to voltage by the motor velocity constant: https://en.wikipedia.org/wiki/Motor_constants

Although it's fundamental for all motors, the actual constant is arbitrary for each motor. Voltage is still proportional to change in field is proportional to speed; at zero voltage you have zero speed.

A large dead AC load has its own speed- zero. It's connecting the engine without a clutch, but the engine makes hundreds of megawatts. The 6' wide generator wants to go from 3600 RPM to a standstill in a few inches. A full speed rotor can have up to 5 gigajoules of rotational inertia; over a ton of TNT.

On a live grid you've got a synchronoscope: https://youtu.be/LVozbX9Mn1c?si=RhRPIkuTVxh4Wseq&t=181

Which is great, because connecting it backwards would cause a slow motion explosion. On a dead grid the only thing saving you is the resistance in the wires; it would be LOUD if you connected to it and you would very likely increase the ventilation through the walls of the plant.

Re:Microsoft blames monster email outage on software defect ... by jeffreygoesto on Tuesday November 17, 2026 @02:44AM (#50528603)
Somebody sent on HN a while ago how a small water power plant is synced to the net [0], I found it super fascinating.

[0] https://youtu.be/xGQxSJmadm0?si=fVdJXtN0bCBX_fyW

Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @04:13AM (#50529178)
That's for a different type of motor. Synchronous motors always run at line frequency no matter the voltage; induction motors always run at a bit less than line frequency. And stepper motors run at the step rate, of course they're not directly connected to the grid because that wouldn't work, but they are also voltage-independent motors.
Re:Microsoft blames monster email outage on software defect ... by arghwhat on Tuesday November 17, 2026 @12:40PM (#50535118)
nit: all motors follow their fields, and no motor is implicitly tied to mains grid frequency.

That we often bypass local field control and tie large industrial synchronous and asynchronous (induction) motors directly to grid is narrow and quirky special case that is common for historic raisins but really quite awful, with staged delta-star or dahlander winding relay tricks to go from insane start currents down to just absurd.

Brushed motors are just synchronous motors with a primitive mechanical field control attached, allowing them to reduce the field rate in response to load.

Re:Microsoft blames monster email outage on software defect ... by account42 on Wednesday November 18, 2026 @07:02AM (#50545256)
> historic raisins

How long do you need to dry a grape for it to not only become a raisin but a historic one?

Re:Microsoft blames monster email outage on software defect ... by AlfredBarnes on Tuesday November 17, 2026 @08:24AM (#50531033)
ima use this "increase the ventilation through the walls" for when people put holes in things. It's great.
Re:Microsoft blames monster email outage on software defect ... by QuantumNomad_ on Tuesday November 17, 2026 @11:10AM (#50533483)
It’s definitely up there somewhere near to “unscheduled rapid disassembly”.
Re:Microsoft blames monster email outage on software defect ... by trollbridge on Tuesday November 17, 2026 @08:45AM (#50531300)
Note that going to solar and power storage fixes a lot of this since it has far more control over synthesising the frequencies needed, as opposed to big spinning turbines with lots of inertia.
Re:Microsoft blames monster email outage on software defect ... by lowkeyoptimist on Tuesday November 17, 2026 @10:01AM (#50532300)
The Iberian Peninsula blackout last year would like a word...
Re:Microsoft blames monster email outage on software defect ... by bdamm on Tuesday November 17, 2026 @12:41PM (#50535139)
Ah, no. Solar and wind are great energy sources, but electronically synthesized sine waves (what you get out of a solar inverter) can also have feedback harmonics that can build into destructive frequency waves. It's not a huge problem as long as the majority of generation is backed by rotating mass, but increasingly power islands and ever more distributed solar generation is tipping the scale of that equation, so we're likely to see more of these sorts of distribution and transmission issues caused by poorly behaving frequency generators' emergent behavior.
Re:Microsoft blames monster email outage on software defect ... by aforwardslash on Tuesday November 17, 2026 @12:54PM (#50535352)
Actually, its the opposite; afaik, injecting sythesized ac requires an already running ac system so it can match the frequency. Its non-linear characteristics can cause catastrophic imbalances in the grid - see portugal/spain blackout.
Re:Microsoft blames monster email outage on software defect ... by eldaisfish on Tuesday November 17, 2026 @10:29AM (#50532717)
this is mostly irrelevant to synchronous generators and operating the power grid.

In a synchronised power grid, machine speed is connected to electrical frequency, that's it.

Voltage and reactive power are controlled by the strength of the rotating field, not speed.

Re:Microsoft blames monster email outage on software defect ... by andix on Monday November 16, 2026 @08:01PM (#50526000)
External load can be blocked/limited extremely easily for an internet service. Just block all incoming connections or limit ip ranges at firewall level. Load can be incrementally increased by unblocking more and more ip ranges.

(It might be better not to use CIDR ranges, but use the lower bits instead if possible. For example starting to unblock all IPs that end with .80 until .90, to get more random geographical distribution)

Which leads to my suspicion it was internal load that brought the system down. Communication between the servers (like delivering queued up messages from one O365 account to another), and not external access from users and apps.

Re:Microsoft blames monster email outage on software defect ... by dboreham on Monday November 16, 2026 @08:57PM (#50526426)
Quick note that sometimes it's not that simple. I've worked on a somewhat similar system (although overall much smaller in scope). What we found was a) clients hop between networks and hence IP addresses quite frequently: mobile devices. And b) with sync type protocols you can't just randomly serve whatever sliced and diced subset of the incoming traffic you like. What happens is that many devices end up never achieving current sync state, so they re-try. In our service we had to implement a protocol-aware, and device ID-aware filtering scheme. It had the property you mention where you could dial the proportion of load up or down, but it did it across the known population of devices: if a device got white-listed at t=0 it remained white listed for the remainder of the day. Once it achieved up to date sync it got put into another load bucket. The bucket of devices transitioning from out of sync to synced up was what we limited. It was only after we implemented such a scheme that we got past our thundering heard / can't bring the service back up after an outage problems.
Re:Microsoft blames monster email outage on software defect ... by andix on Monday November 16, 2026 @09:29PM (#50526655)
I don't get this issue. The vast majority of clients would not change IP during a short sync that takes maybe a few seconds. And a switchover from an unblocked to a blocked IP would either result in a partial or in a failed sync, without any consequences.

If the load balancer already does some connection tracking, it can just keep serving existing connections and only drop new ones. If connection tracking is not available, the IP blocklist would be a suitable option.

Re:Microsoft blames monster email outage on software defect ... by mike_hearn on Tuesday November 17, 2026 @03:31AM (#50528907)
Clients that wake up and sync can get different IPs on every sync on some networks because of carrier NAT.

IP blocking is generally not what you want though.

Re:Microsoft blames monster email outage on software defect ... by andix on Tuesday November 17, 2026 @08:45AM (#50531288)
> IPs on every sync on some networks because of carrier NAT

Okay, they get different IPs during the day, fine. But it's still kind of randomized. Stepping through IP ranges will reduce load and give access to more people.

I really don't get the issue you are seeing with this.

Re:Microsoft blames monster email outage on software defect ... by psd1 on Tuesday November 17, 2026 @08:55AM (#50531418)
IP blocking is dirt cheap on your edge routers. Application-aware binning needs at least your app gateway to have spare capacity. I can see that going tits-up.
Re:Microsoft blames monster email outage on software defect ... by tremon on Tuesday November 17, 2026 @07:37AM (#50530583)
> Which leads to my suspicion it was internal load that brought the system down

IIRC Microsoft posted a comment around 10pm where they discussed rate-limiting external connections, and many reactions on last week's HN thread were like "wait, you so didn't even drop load before now?" So internal load may have been a contributing factor, but it's by no means certain given that they didn't stop serving clients.

Re:Microsoft blames monster email outage on software defect ... by drusepth on Monday November 16, 2026 @08:11PM (#50526086)
> In practice black starts are extremely rare, but they can happen after a cascading failure in which something causes some generating stations to trip, redirecting load onto the remaining stations which can't handle it causing them to trip in turn. The entire grid can collapse in just 60 seconds or so once that kind of runaway failure starts.

This is absolutely terrifying knowledge to know given the competency of the current administration and their attitude/response to recent "black swan" outages.

Re:Microsoft blames monster email outage on software defect ... by water42 on Tuesday November 17, 2026 @12:05AM (#50527675)
why did you feel a need to explain cold starts, a standard distributed systems problem, with an elaborate story about "black starts" from an analogous space? you can just explain what cold starts are directly and talk about the relevant problems that come up when doing so.
Re:Microsoft blames monster email outage on software defect ... by jdironman on Tuesday November 17, 2026 @12:12AM (#50527704)
i found their comment useful as someone who works in tech but doesn't always understand things at scale like what they explained. So it increased my understanding and actually made me think about some of my own work from a different angle versus say reading a comment critiquing using analogies.
Re:Microsoft blames monster email outage on software defect ... by Scharkenberg on Tuesday November 17, 2026 @12:21AM (#50527756)
It was quite helpful. What are you, the explanation police?
Re:Microsoft blames monster email outage on software defect ... by water42 on Tuesday November 17, 2026 @03:04AM (#50528726)
Readers would understand what happened better given a distributed systems example. Lots of problems come up cold starting distributed systems that aren't explainable through "black start"
Re:Microsoft blames monster email outage on software defect ... by balarjin on Tuesday November 17, 2026 @11:12AM (#50533516)
> Readers would understand what happened better given a distributed systems example.

Readers who know about distributed systems and their failure modes already understand the thing being explained. They (and you) are not the target audience. Let people who are not already experts learn something by analogy.

Re:Microsoft blames monster email outage on software defect ... by mike_hearn on Tuesday November 17, 2026 @03:37AM (#50528948)
Well, mostly because it just occurred to me and I thought it would be worth sharing.

But the analogy is useful in another way. The electricity industry plans and rehearses for this kind of thing, even though it's impossible to simulate. I used to be rather skeptical that these rehearsals would be effective but Portugal black started last year after a solar plant in Spain malfunctioned and took down the whole Iberian grid. It took almost exactly 24 hours despite the grid collapsing several times during the process, which is the usual estimate for how long things will take. So it seems they have this stuff well under control.

I won't slam Microsoft too hard for this outage because this type of scenario has played out at many companies over the years. Yet I never encountered an ops team that has written procedures for how to recover when the entire service has entered a crash loop under peak load.

The exact analogy we use for this doesn't matter much as they are all pretty evocative, but using black start has the advantage of reminding us that there's another branch of engineering where this necessity is taken seriously and rehearsed. They also engineer for it ahead of time so they can prioritize customers like hospitals and airports under overload conditions. I think it would be a good feature for cloud LBs to be routinely differentiating between e.g. customers on paid plans vs free so they can serve paid customers first if there aren't enough live backends.

Re:Microsoft blames monster email outage on software defect ... by water42 on Tuesday November 17, 2026 @04:01AM (#50529125)
>Yet I never encountered an ops team that has written procedures for how to recover when the entire service has entered a crash loop under peak load.

this is an interesting observation, since this should be table stakes for running a high scale distributed system. there are plenty of companies that do this and write about this extensively. it definitely becomes less important until you get to certain levels of scale though - a random app with 10k users has a different standard than, e.g., google cloud storage.

Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @04:19AM (#50529209)
Was that outage shown to be a single solar plant's fault? IIRC it was actually some kind of emergent grid oscillations.
Re:Microsoft blames monster email outage on software defect ... by mike_hearn on Tuesday November 17, 2026 @04:44AM (#50529372)
Like in any such failure it's an onion you can unwrap several layers deep. The initial cause was a solar plant where the inverters malfunctioned and started dumping unsynchronized power into the grid in an unobservable way (grid operators didn't know where it was coming from). Their attempts to fix the issue caused other problems which combined with some plants being too sensitive and tripping when they perhaps didn't need to.

At the time Iberia was relying >50% on renewables which is very dangerous. The operators tried to recall gas power plants to provide inertial stability, but the plants were cold as they'd been taken offline in the morning to make way for solar (prioritized due to green politics), and even though gas plants can spin up very fast compared to other kinds they could not warm up fast enough to rescue the situation as it was all over thirty seconds later.

So it depends how you define "fault" but yes the starting point was a broken solar plant. Thermal/hydro plants stay locked together through physics because there's a prime mover connected to a generator, but solar plants rely on computer-managed simulations of the rotation that tries to follow the rest of the grid. When you have more simulation than reality it can get tricky fast.

Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @04:52AM (#50529417)
You seem to be going quite far out of your way here to shit on the free energy machines.
Re:Microsoft blames monster email outage on software defect ... by mike_hearn on Wednesday November 18, 2026 @04:06AM (#50543808)
No, that's just a neutral writeup of what happened.

Less neutrally, there have been far too many years of people pretending that you can build a grid out of 100% renewables. It was a false claim from the beginning, but engineering was overridden by ideology. The Iberian outage was one of the consequences - unfortunately merely a preview of coming attractions, as our political class is in denial and would rather believe lies than address the truth that it's caused by their own laws. Now it's only a matter of time until the entire UK or European synchronized grids experience a collapse of their own, with a similar root cause. We can't say we weren't warned.

Re:Microsoft blames monster email outage on software defect ... by cowlevel on Wednesday November 18, 2026 @06:30AM (#50544997)
yeah you're definitely posting a bunch of ideology
Re:Microsoft blames monster email outage on software defect ... by ray_v on Tuesday November 17, 2026 @12:38AM (#50527838)
So is it weird that there have been three cascading failures in about as many days ... Almost as if we're being probed
Re:Microsoft blames monster email outage on software defect ... by Epa095 on Tuesday November 17, 2026 @01:27AM (#50528105)
Its only analogous in the sense that we must have capacity ~>= load, and that you can get cascading failure. Where the analogy breaks down is that in power production you also have that capacity ~<= load, so you must have load and capacity roughly balanced. But that's not the case for the outlook servers, nothing explodes if there is 1000 over-provisioned servers.
Re:Microsoft blames monster email outage on software defect ... by pyrale on Tuesday November 17, 2026 @03:12AM (#50528774)
If we are to add to the power grid analogy, black starts are so rare because there is a lot of work going into mitigating failures.

For instance, a generator failure that cannot be mitigated by dropping load will probably result in islanding the grid: circuit breakers trip some lines and your network becomes several sub-networks. The sub-network that tripped can then be re-powered without a black start.

Re:Microsoft blames monster email outage on software defect ... by wvh on Tuesday November 17, 2026 @05:27AM (#50529661)
In my experience, we too often try to engineer around unavailability with caches, queues and cold serving, instead of just telling clients they can't have the cookie and have to wait. Maybe LBs should have a random drop policy in case of extreme overload of backend services, enough to keep things working somewhat both internally and visibly for clients, instead of contributing to the stampede and the whole system subsequently going completely down for all. "Too many" is a very real state.
Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @08:17AM (#50530959)
Google SRE Book - Handling Overload: https://sre.google/sre-book/handling-overload/

An important part is that clients pre-fail most requests and don't send them if the server is overloaded. They just assume the server would return an overload error.

Re:Microsoft blames monster email outage on software defect ... by trollbridge on Tuesday November 17, 2026 @08:48AM (#50531334)
This has been understated; at Google everything had to be engineered for a cold start. And they’d actual go test this periodically, using techniques to do so so end users never knew.
Re:Microsoft blames monster email outage on software defect ... by greggsy on Tuesday November 17, 2026 @07:35AM (#50530554)
I too have played Satisfactory
Re:Microsoft blames monster email outage on software defect ... by mike_hearn on Tuesday November 17, 2026 @10:18AM (#50532531)
Hah, actually I never played it. But it's cool to know it simulates this.