Re:Microsoft blames monster email outage on software defect ...
by nilptr on Monday November 16, 2026 @05:37PM
(#50524461)
There's literally no such thing as "black start" at Microsoft that I am aware of. The way services are brought online initially is through a build out process coordinated through hundreds of Teams. It's an organic, messy, and painful way of doing things. As such, build outs are a human driven process with bits of automation spanning sometimes 20+ years of engineering touches, troubleshooting guide, and some additional tribal knowledge thrown in passed down through the years. There is little consistency across teams and sometimes even within a team as every Azure Cloud is effectively slightly "different".
[ Parent
| on HN ]
Re:Microsoft blames monster email outage on software defect ...
by markus_zhang on Monday November 16, 2026 @10:43PM
(#50527158)
Reading this, I can literally hear people shouting over the mic.
“Ready? 10 seconds to power on server alpha-romero-one-o-four!”
“Load—at five percent, check! All systems stable!”
Re:Microsoft blames monster email outage on software defect ...
by nilptr on Monday November 16, 2026 @11:18PM
(#50527379)
Ha, no. It's not quite like that, but it can take weeks to bring up a service especially if you're in the lower rings. Often time core cloud stuff is being built out while services are also building out so stuff gets stubbed out and then you have to go back and slowly fill in the dependencies.
It's chaotic and stressful in a slow asynchronous way.
Re:Microsoft blames monster email outage on software defect ...
by markus_zhang on Tuesday November 17, 2026 @08:45AM
(#50531283)
I seem to get some morbid pleasure out of it.
Re:Microsoft blames monster email outage on software defect ...
by bitwize on Tuesday November 17, 2026 @11:26AM
(#50533782)
"Type 'cookie', you idiot. I'll head 'em off at the pass."
"Rabbit, flu shot, someone talk to me!"
Re:Microsoft blames monster email outage on software defect ...
by coliveira on Monday November 16, 2026 @11:50PM
(#50527600)
Nobody should ever depend on things done by Microsoft. It's in their culture to create things this way, it is totally undependable. My guess is that we still haven't seen the complete mess that MS can create with their online services, and everyone will pay for this reliance.
Re:Microsoft blames monster email outage on software defect ...
by SV_BubbleTime on Tuesday November 17, 2026 @12:51AM
(#50527908)
Sounds like the should reduce their domestic workforce and run more AI and H1B.
Re:Microsoft blames monster email outage on software defect ...
by hobo123 on Tuesday November 17, 2026 @01:33AM
(#50528146)
Something similar to Brooks' law - adding more H1bs to a project makes it later?
(I have no experience with H1Bs nor beef, I'm a German who's merely had the joy of seeing failed offshore projects.)
Re:Microsoft blames monster email outage on software defect ...
by psd1 on Tuesday November 17, 2026 @05:28AM
(#50529664)
Sure, but also, leftpad.
Re:Microsoft blames monster email outage on software defect ...
by gosub100 on Tuesday November 17, 2026 @07:12AM
(#50530362)
It's so bizarre they have no consideration for UX, or reliability or quality. But they will keep 16-bit APIs for backwards compatibility to DOS because some peanut butter factory in Indiana still uses it.
Re:Microsoft blames monster email outage on software defect ...
by trollbridge on Tuesday November 17, 2026 @08:46AM
(#50531313)
I’m not aware of Windows having any 16-bit DOS APIs since 32-bit Windows 10 was retired.
Re:Microsoft blames monster email outage on software defect ...
by calvinmorrison on Tuesday November 17, 2026 @12:06PM
(#50534519)
I'm in the middle of upgrading software released right before the invasion of Iraq. Things chugging along great
Re:Microsoft blames monster email outage on software defect ...
by water42 on Tuesday November 17, 2026 @12:06AM
(#50527678)
you're telling microsoft doesn't think ahead about having to cold start all services within a product during a worst case scenario outage?
Re:Microsoft blames monster email outage on software defect ...
by jacquesm on Tuesday November 17, 2026 @12:29PM
(#50534953)
I'm sure none of those people were amongst the 60,000 or so targeted in the recent round of employee reductions.
Re:Microsoft blames monster email outage on software defect ...
by anankaie on Tuesday November 17, 2026 @01:59PM
(#50536440)
There is a BCDR strategy for each service at different tiers (based on which "ring [of criticality]" your service is in). Every service is supposed to pass through a (yearly? biyearly? don't remember) compliance certification. Not sure which rings require practicing it, though.
All of it is documented through a unified catalog of services and their associated metadata.
Source: Worked in MSR and shipped services through Azure / CoreAI.