6 hours ago they seemed to not exactly understand what the problem was, and that was 6 hours into the outage.

This is concerning.

all 3!

Yes.

That’s why it’s taken over 12 hours now, no one’s left that knows without copilot’s help.

Lately it seems like more non-techies are souring on the company.

Top notch user experience peak was Windows XP and every subsequent release was massively larger, clunkier and slower while requiring way more resources, was spying on the users more and more. Office has similar story, peak of usability was maybe 2007 with ribbon introduction (which still a lot of power users hated with passion since it was far from clean design, more seemingly designed by committee of marketing guys).

which, besides hatred, manifests in malice, like Nightmare Eclipse. I wouldn't be surprised if this outage was an engineer leaving a parting gift, using a bug they probably knew of for years but management didn't allow them to fix.

We're... how many decades in? When I still did SRE work, I was pulled into no fewer than three dozen incidents because outsourced labor confused assets from TICKET-123 with those from TICKET-321. Target avoidance plays a part, and as layers are put on, mistakes become more likely. I guess it's a form of an XY problem, too.

Enterprises have been failed time and time again by poor MS products, but techies will steadfastly refuse to move off it, so the business has no option.

My suspicion is that lots of the companies are so attached to all the MS stuff (and that's what their staff know), that it would be an incredibly large lift for them to move off it.

Like, I'm at a very windows/azure focused shop now (for the first time in my career), and literally everything is already on MS, or moving towards MS.

There are some good reasons for that, and some bad reasons, but I suspect that most of the sales process happens at the executive level, rather than the technical level (and Excel runs the world, so why not let MS do everything else?).

One guy but it's Stephan T. Lavavej the guy who the "STL" was literally named after.

I work at a company with a few thousand employees in the Netherlands, and have been moved around a few teams in the last 7 years (not by choice, the team is nuked and I'm shuffled off elsewhere).

Technically, the product is supposed to have a "handover" to another team who can do the maintenance. What happens in practice is the person who worked on it gets saddled with maintaining it forever along with taking up the responsibilities of the newer work that comes along their way.

I've accumulated a string of such projects, including one that's used extensively by a few large companies, and the only person maintaining it is drumroll ... me, in whatever time I can spare for it.

Imagine if say BMW decided they don't want to manufacture their motors, suspensions, drivetrain etc. Outsourcing 100% of that knowledge to external suppliers, for all models, from now on forever. How do you think market would reacted to such move ie by valuation of stocks?

[1] https://www.youtube.com/watch?v=y_PrZ-J7D3k

But the biggest fear is how is ai going to improve then ? We’ve seen it can’t feed itself on data it produced, the model quality decrease. But if there is no more human to produce ever better quality code because they rely on llm to generate code, we’re going to reach a wall.

Microsoft can't fix Outlook because they don't have Claude, and Anthropic's employees can't fix Claude because they can no longer log in without Outlook?

I started digging into the historical blog posts, GitHub comments, etc. and I realised everybody at MS who'd been working on them, SWEs, PMs, architects, etc. had either left the company, or moved to other teams. And the one constant about the internal moves was that people were now working in some "AI" related project or product. Literally, I didn't find a single person who wasn't now working on AI.

Clearly, there has been a huge internal push to get into AI. Resources have been reallocated, and anything non-AI related has been starved. And since working on an AI project is probably considered the only route to advancing your career, anybody with any ambition has moved to a team doing something AI-related. The result is that huge numbers of products and teams doing vital but non-AI related work have been hollowed out, and now we're seeing the consequences.

But no, read Hacker News or other niche places enough and eventually you realize a lot of the things we all know or use was a solo effort or a very small number of people, sometimes not even an assigned project but something they did on the side. Cue the xkcd about the lone open source project maintainer holding up the entire modern web but apply it to the tech giants and there you go, it happens there too.

Lots of its features - App Services, DNS, managed DB, Front Door, Blobs, Durable Functions, Service Bus, App Insights - are very powerful. The frontend portal is mostly fine for management. The Az CLI makes it easy to provision things quickly. Coupled with .NET Core on MacOS, I've not needed to touch Windows locally for years.

I could learn AWS or GCP, but that's just going to cost more time when I'd rather use what I already know to build more things. With Azure, I'd hate to see the rot set in further.

"When in danger, when in doubt, run in circles, scream and shout".

If you lose 90% of developers to AI, but everything breaks, how much money have you saved?

The underlying moral issue is that the management and money people have contempt for people who don't manage and don't have money, including both customers and enginers.

It's not just an inhumane system - although it is, that, brutally - but it's also technologically dysfunctional.

Sorry, had to.

We're doing anything to avoid single points of failure in our products, but our IT is doing the opposite.

The idea of paying for a backup system, or even having one on standby for situations like this is just not on people's minds, or if it is, it's simply not affordable because it would cost you too much compared to your competition.

Razor thin margins and generally getting away with it is what makes this work. Until it doesn't.

Sure, it makes sense for mission critical companies (utilities, emergency services), but I don't think it does for an average company. Especially that you don't take much blame if everyone is down.

We had a situation recently where cellphones were down, we switched to landline.

You don't need a 1 to 1 mode backup for everything, just a failover plan.

Unless your company is infrastructure or medical, I doubt they have a backup/failover for loss of power. Or water.

Also, multiple bankers I'm speaking to about Anthropic's IPO can't send or receive work e-mails. So that's a thing.

replaces all occurances of microsoft/office 365 with microsoft/office 362 (not in links, just in the visible text)

[0] https://addons.mozilla.org/en-US/firefox/addon/microsoft-362...

https://techcrunch.com/2026/01/05/microsofts-nadella-wants-u...

We just don’t know how good we’ve got it, that’s all!

At my day job, we have a calendar integration that polls Microsoft's Graph API across hundreds of Microsoft 365 tenants. We saw our first metric spike in 5xx errors at 2026/11/12 17:33 UTC. As of writing it's 2026/11/13 09:44 UTC and we are still encountering a full outage, so 16 hours and 11 minutes.

With a SLA calculator [1], assuming no other downtime this month and entering `16h 11m 0s`, we get: * Yearly SLA: 99.82% * Monthly SLA: 97.78%

I found an SLA document list here [2].

I downloaded Service Level Agreement for Microsoft Online Services (WW) English (2026/10/01). There's a heading for Exchange Online that says:

Uptime Percentage : Service Credit < 99.9% : 25% < 99% : 50% < 95% : 100%

Which seems to agree with what the original commenter said for this month.

[1] https://uptime.is/reverse [2] https://www.microsoft.com/licensing/docs/view/Service-Level-...

These targets feel very very low nowadays.

99% uptime for email means 3.5 days/ year no email.

(although 99.9 = about 8 hours/year would seem bearable - but maybe not for a company like microsoft)

Depends on how it appears, if it was one or several large chunks that is much more disruptive than minutes per day. 3.5 days/year is equivalent to less than 15 minutes/day.

It sucks no matter how you slice it, but the severity/impact depends on more than the aggregate downtime.

What could go wrong, said all of the hyperscalers. Enjoy your multi day downtimes.

It's just a wild theory, but maybe there are still too much employees left, standing in the way of AI? I would try firing another 60,000 first.

Microsoft has a moat, but their moat is having billions of dollars to spend on data centers. It's a failure everywhere else. It'll continue to rake in billions in revenue because people need compute now, but it's rotten at the top.

"Have you tried firing all the white men that invented everything and replacing them with people from third world countries where people shit in the street?"

was hilarious.

They absolutely did hire anything but.

We're not white. Not sure why I'm getting downvoted by merely reporting I found an obvious joke funny.

Or did you think that tweet was actually serious advice from a concerned samaritan?

No Outlook, they should make this a public holiday!

I am looking at other current events and beginning to wonder if there is some kind of correlation.

The biggest issue has been so many security and approval flows depend on email, so all processes have ground to a halt.

Government I have no idea, but loads of businesses don't use Outlook - alternatives do exist. I've personally not used Outlook for well over a decade now.

Quick google AI mode check says:

- MS 365/Outlook: ~40%

- Google Workspace: ~25%

- Zoho: ~2%

And then a bunch of others - Rackspace, Zimbra, IBM WebSphere Lotus Notes Blue For Enterprise Service Bus Watson Edition et al

Exchange and Active directory are the biggest obstacles by far, bureaucrats can learn how to work with open source office alternatives easily, and underlying OS can be anything as long as you have few icons to click on.

I see that you're a hater. Outlook sucks but your solution is to cure cancer (Outlook) with Alzheimer's (Zimbra, Lotus Notes)?

And in terms of classic 'corporate' sectors like law, finance, etc, it also close to 100%.

Whereas in "tech" it is far more likely to be Gmail.

I _still_ get people who are non technical that work in 'corporate' sectors that can't join a Google Meet because they think it is Teams. They literally haven't ever used a Google meeting before.

I love this kind of wording. It makes everything suddenly more serious lol.

It would not surprise me if the entire project is complete and utter (human-written) slop

I was using a dedicated VPS, maybe that was part of the issue, maybe self-hosting from a residential IP might work better. But it was easier for me to let iCloud handle my domains email. It's been flawless ever since even though they're not really a big player in e-mail. Used Proton for a bit, that was (almost) flawless as well, though even Proton occasionally gets blocked by stricter corporate rules.

MY University's whole email system is Outlook/365, and that also seems kind of dumb to me.

My guess is that having duplicate system would cost (in money for double subscriptions and lost productivity) much more than the benefit of being available 0.01% more. Especially for universities where almost nothing is really urgent and can't be delayed by a day.

frog boiling means that many never noticed how big (and therefore unspecialised) microsoft was until it was too deeply integrated

BBC is running live coverage of it at the level of government and ministers.

I don't expect a fundamental shift but the boss having no access to his mails seems certainly like something I would bring up quite a bit.

It will help a bit, but it would need much more than 24h to have a real impact. If their service is still down in a week, now we're talking.

Barring corona & year end holidays, this is probably the closest we can get to a global work holiday.

This means you cannot do anything that requires email auth (OTP, signup etc)

It is much easier to read the code you wrote.

It's such bedrock infrastructure that I've seen it used as fallback in the event of electronic trading infrastructure outage, brokers would accept specifically formatted emails as orders.

At this point it would probably be less impact to have phones go out globally because people would fallback to walking across the floor, using zoom/slack for simultaneous communication or email for more async.

That's why most don't realize how big GMail is. Google has 1.8 billion GMail users, including a great many who are using their Google email professionally (no matter if it's a Google Workspace with their own domain or a redirection or just a plain GMail). Apple has 1 billion email users.

Microsoft 400 million and, granted, many businesses but it's not a fourth the number of email users GMail has.

And many, many, many people (real estate, delivery drivers, Uber and the likes) are using either Apple or GMail email from... Their phones.

It's a good thing it's only the "small" Microsoft email users base that is affected and not the 1.8 billion GMail users.

https://gioorgi.com/2020/mail-server-on-docker/ https://github.com/docker-mailserver/docker-mailserver

It is very easy to set up (just a bit of time on average every 6 month to follow upgrades) and work out of the box.

Europe outlook users are experiencing problems too.

Quick lets collect all options people have to self host! in addition to docker-mailserver im aware of

- https://github.com/stalwartlabs/stalwart

- https://github.com/mailcow/mailcow-dockerized

Geez, that's right! :-D

It seems to be a fairly popular project though, so what does that say about the correlation of README.md quality to success?

In the end I paved over my Linux VM with OpenBSD and used Dovecot+OpenSMTPD which I found dead simple and so far faultless. I have nothing but kind words for it.

[0] https://stalw.art/docs/configuration/

Nowadays my interests have skewed towards being able to have groupware features without much maintenance so groupware in a box suits my needs just fine. The learning curve to doing serious customization with a large FOSS groupware project might be a bit longer but that is offset by other factors.

Some problems are not solved by individual choice. If 30-40% of your important contacts use service is down, your ability to work goes down dramatically.

This is my biggest complaint when self hosting. You do everything right and they still send your mail to junk or just silently discard it while telling you 250-Ok for "reasons".

Microsoft have already recently kicked out self hosters from their "Smart Network Data Services" portal by only allowing the ASN owner (ie. your cloud provider) to access it.

https://www.centennialsoftwaresolutions.com/help/transcript-...

This will only make it harder for Microsoft to pull all customers to their cloud offerings. I think in general that's a significant blow to SaaS-ification of the field.

Do they lose any significant number of customers? Do anybody in leadership get fired?

Do reading the twitter status updates give you the impression of an organization that takes responsibility seriously?

Or do it sound more like an organization that is optimized for evaporating responsibility?

I think this will change nothing.

And I don't exclude myself here by the way. I know what I'm doing, I know what my colleagues are doing, I know what my managers are doing. We're all pretty busy, but our productivity is very low.

>Do reading the twitter status updates give you the impression of an organization that takes responsibility seriously?

Not implying otherwise, to be clear. Very far from defending MS.

Did you personally select anything IBM lately? At some point many years ago, people were saying "Getting rid of IBM in a corporate setting is close to impossible.

Anyone want to guess if any of those afflicted _could have_ caught an issue that may have prevented this disaster?

2) they’re only expecting 10% or so to take the buyouts, so more like 6,000

I mean it's e-mail.

Its all connected

M365 is a suite. Make a 365 email and you also make an account in Entra, connections, onedrive, sharepoint, teams etc etc etc

Its not "just email"

They just one day decided to rename it Skype for Business while making it incompatible with Skype because brand synergy.

Teams - absolute garbage OneDrive - clunky garbage Sharepoint - massive pile of shit

I'll spare readers the full anecdote, but I worked for the company running the largest mail service in the southern hemisphere, and at the same time a few of the large mail services in the northern hemisphere in the 1990s. We met with the Exchange team who wanted us to adopt their product. Where we could run a given service on 2-3 servers, Exchange sized by MS required 300 of them for just 1 of our countries. Fortunately our cash constraints prevented us going down that road. They offered the license for free and exerted pressure on the C-level, as you can imagine there were heated conversations and some choice language between mgmt and engineering on this topic.

Thanks to AI, heated conversations between management and engineering is mostly a thing of the past. Unless the engineer wields an outsized influence, they will be shown the door and replaced with a lower-priced AI-augmented engineer. Which might explain the increasing frequency of shitshows we are seeing lately.

No.

Because of the infinite resources premise and because there exist email services of that scale that are reliable. Email is not an unsolved problem.

The rest of your comment does not make much sense to me ...

Jesus, make sure you stay away from signal processing if this trickle of data scares you

Just for a sense of scale, on a single CPU core that is something like 1M instructions per email. ONE CPU CORE.

Internal software goes through rigorous change control, change windows, change freezes around key business dates, etc to minimize risk.

Then you sign up for some SaaS and they blow out email (which aside from security needs 0 updates lets be honest) midweek, mid business day without warning or reason.

Trillion dollar companies shouldn't even be able to have excuses.

You have infinite money, stop fucking up.

Cash can't fix culture.

hard to change the minds of a bunch of pissed off folks when your political campaign funding is pushing AI ads about transfolk and not grocery prices or datacenter rollouts.

https://en.wikipedia.org/wiki/XBill

Isn't there an old saying about putting all one's eggs into one basket?

It's simple really, I've no cloud accounts, and even my Android phone is de-googled. I'd never trust any data to the cloud and least of all to MS.

What's happened here with MS clearly illustrates one of my reasons. I find it very strange that so many use Onedrive, Office 365, etc. and trust them even though they've not one iota of control over their operation.

Given the nesting of services at MS, the fractious org structure, and the length of service of some of these systems it was already a bit of a tinder-house.

AI just provided the accelerant, and while I don’t work at MS I can imagine they’ve lost some serious tenured talent over the last 5 years. I’d be surprised if anyone is left that actually understands how it’s all glued together.

Or rather, that might be the case before this outage. I suspect the org has relearned a fair bit over the last day.

But even this requires someone high up in the organization to have the insight that reliability requires some level of “inefficiency” — ie retaining certain highly paid staff, duplication of effort, over provisioning of resources, etc.

The irony is that the execs who could make those decisions are convinced that LLM tools obviate those needs, and shift all spare resources in that direction. But without heavy constraints from experienced humans, LLM tools will only multiply the complexity at the root cause of these sorts of outages.

I'm an outsider who used to work at a different cloud provider, so take this with a grain of salt, but this doesn't smell like a limits and provisioning problem, per se. It's not necessarily like they just had 1 too many emails tip the scales for the entire world. Until today stuff was running okay. This was a monitoring and fail-over problem, at a comically tragic scale - a cascade failure. Architecturally they obviously did not design for a world where they could damage control failure at this scale. Failure should be expected and designed for at literally every layer. They cheaped-out on careful design in order to play catch-up building a public cloud offering and here we are.

> We determined the elevated load isn't driven by an increase in traffic, but by a number of servers being unexpectedly unavailable; straining the remaining infrastructure. > We found a core process on affected servers is becoming overwhelmed, causing capacity loss. > We’re identifying alternate workstreams that can be applied to increase capacity in the face of sustained traffic management imbalances. Hmm maybe they can migrate their infra to a hyperscaler to prevent capacity related outages like this in the future while working on a fix.

> Our investigation indicates the failed health checks may originate from a specific component that's responsible for processing server health. Someone prompted "health check not healthy please fix MAKE NO MISTAKES!!!".

> we're moving additional traffic off the affected infrastructure, and we're restarting affected servers to return them to service > In parallel, we've initiated a restart of all affected servers and are monitoring health as they return to service. > We're also preparing automated recovery systems to continually detect and restart unresponsive servers, to maintain service stability while this change progresses. > We're actively restarting and validating the affected servers to return them to service. > To validate a potential recovery, we're isolating a portion of infrastructure, restarting affected components Jeez, have you tried turning it off and on again?

It seems that not just the engineers but also the people responsible for incident communications were replaced by AI.

Hopefully some of their "customers" learn a lesson and switch to a software without this side effect in worst case. There's plenty of alternatives (LibreOffice, OpenOffice, etc.)

This is not related to this outage, right?

16 hours and nothing...

My email is still working, has been solid for moe than a decade. My calander ap opened as normal this morning. My chat windows are all alive. My digital life is just normal today... with rather fewer emails in the inbox. DROP MIRCOSOFT. Pick up linux. Pay for a proper email service. There is a larger world out there!

If it is indeed the case, this might be the first clear "win" for those who have been warning about the dangers of pushing unreviewed code into production.

Microsoft imposes all of this shit on the world. Poorly thought out products, buggy crap. Its finally come home to roost.

Other source: This is related to a BGP hijack of at least 87.128.0.0/10 belonging to Deutsche Telekom (AS3320) hijacked by AS212416 (Proximity, Bulgaria) passing through AS57344 (Telehouse EAD, Bulgaria) starting at 0633 UTC today.

The final thing we did was photo a QR code with our phones that took us to an Office 365 form to evaluate the course.

I was surprised to see some fields on my form showing values that my course comrades had entered on their phone.

Could the course evaluation have been incorrectly developed and deployed using AI on Office365 by the course developer?

Or was some session state all messed up across parts of Office 365?

I was in a hurry so I decided to come back to check on this form later but now it is just stuck ”Loading…”.

See: Stalwart Mail Server: https://stalw.art/ Bulwark Webmail/Groupware: https://bulwarkmail.org/

Jeez

Note: https://news.ycombinator.com/item?id=50416158