News for hackers. Stuff that matters.
Re:Microsoft blames monster email outage on software defect ... by mike_hearn on Tuesday November 17, 2026 @03:37AM (#50528948)
Well, mostly because it just occurred to me and I thought it would be worth sharing.

But the analogy is useful in another way. The electricity industry plans and rehearses for this kind of thing, even though it's impossible to simulate. I used to be rather skeptical that these rehearsals would be effective but Portugal black started last year after a solar plant in Spain malfunctioned and took down the whole Iberian grid. It took almost exactly 24 hours despite the grid collapsing several times during the process, which is the usual estimate for how long things will take. So it seems they have this stuff well under control.

I won't slam Microsoft too hard for this outage because this type of scenario has played out at many companies over the years. Yet I never encountered an ops team that has written procedures for how to recover when the entire service has entered a crash loop under peak load.

The exact analogy we use for this doesn't matter much as they are all pretty evocative, but using black start has the advantage of reminding us that there's another branch of engineering where this necessity is taken seriously and rehearsed. They also engineer for it ahead of time so they can prioritize customers like hospitals and airports under overload conditions. I think it would be a good feature for cloud LBs to be routinely differentiating between e.g. customers on paid plans vs free so they can serve paid customers first if there aren't enough live backends.

Re:Microsoft blames monster email outage on software defect ... by water42 on Tuesday November 17, 2026 @04:01AM (#50529125)
>Yet I never encountered an ops team that has written procedures for how to recover when the entire service has entered a crash loop under peak load.

this is an interesting observation, since this should be table stakes for running a high scale distributed system. there are plenty of companies that do this and write about this extensively. it definitely becomes less important until you get to certain levels of scale though - a random app with 10k users has a different standard than, e.g., google cloud storage.

Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @04:19AM (#50529209)
Was that outage shown to be a single solar plant's fault? IIRC it was actually some kind of emergent grid oscillations.
Re:Microsoft blames monster email outage on software defect ... by mike_hearn on Tuesday November 17, 2026 @04:44AM (#50529372)
Like in any such failure it's an onion you can unwrap several layers deep. The initial cause was a solar plant where the inverters malfunctioned and started dumping unsynchronized power into the grid in an unobservable way (grid operators didn't know where it was coming from). Their attempts to fix the issue caused other problems which combined with some plants being too sensitive and tripping when they perhaps didn't need to.

At the time Iberia was relying >50% on renewables which is very dangerous. The operators tried to recall gas power plants to provide inertial stability, but the plants were cold as they'd been taken offline in the morning to make way for solar (prioritized due to green politics), and even though gas plants can spin up very fast compared to other kinds they could not warm up fast enough to rescue the situation as it was all over thirty seconds later.

So it depends how you define "fault" but yes the starting point was a broken solar plant. Thermal/hydro plants stay locked together through physics because there's a prime mover connected to a generator, but solar plants rely on computer-managed simulations of the rotation that tries to follow the rest of the grid. When you have more simulation than reality it can get tricky fast.

Re:Microsoft blames monster email outage on software defect ... by cowlevel on Tuesday November 17, 2026 @04:52AM (#50529417)
You seem to be going quite far out of your way here to shit on the free energy machines.
Re:Microsoft blames monster email outage on software defect ... by mike_hearn on Wednesday November 18, 2026 @04:06AM (#50543808)
No, that's just a neutral writeup of what happened.

Less neutrally, there have been far too many years of people pretending that you can build a grid out of 100% renewables. It was a false claim from the beginning, but engineering was overridden by ideology. The Iberian outage was one of the consequences - unfortunately merely a preview of coming attractions, as our political class is in denial and would rather believe lies than address the truth that it's caused by their own laws. Now it's only a matter of time until the entire UK or European synchronized grids experience a collapse of their own, with a similar root cause. We can't say we weren't warned.

Re:Microsoft blames monster email outage on software defect ... by cowlevel on Wednesday November 18, 2026 @06:30AM (#50544997)
yeah you're definitely posting a bunch of ideology