Or, rather, it's acting like it usually does.

I just assumed it was Teams being Teams.

Back on Window Mobile 6.1.

Battery life usage quickly improved. :)

The worst part about Outlook is the broken search. In the world of infinitely smart AI, the e-mail search in Outlook is embarrassingly stupid.

If your org is using Intune or Defender EDR then you probably also are impacted.

Any other service that interacts with Outlook in some way will create downstream effects in other services expecting Outlook to work normally.

The Y axes are entirely independent, and so when there's an outage for Microsoft Outlook (which peaked at ~130k reports) and you get a negligable spike in pepole who get confused and report gmail (~1k reports), the graphs all look correlated and connected.

They need to show the graphs with an absolute y-axis set by the worst outage, or scale the graphs by estimated average daily users or something.

Because, well, once upon a time there was Exchange Server and MAPI (the "Messaging API"), and that actually worked pretty well, and pretty much everyone started using it (or one of its EWS/ActiveSync offshoots), but then the Internet happened and ruined everything, but we got MAPI-over-HTTPS eventually, which by that time had grown so many warts that nobody at Microsoft really understood it anymore, so that got deprecated, even though, well, everyone and everything still uses it.

The new hotness was Azure-slash-Office-365, and that's where the "new Outlook" web interface lived and (sort-of) continues to live, and the on-prem customers got stuck with the "Exchange Subscription Edition", which was basically Exchange 2019 plus security hotfixes at extortionate rates, but with none of the new functionality.

And, since MAPI continued to be really, really hard even in the age of Copilot, access from the new hotness to the deprecated on-prem servers was implemented via IMAP (LOL), but force-proxied via the outlook.com cloud/clownshow.

Which is why, right now, many Microsoft customers that thought they were self-hosting their email are entirely down. Lessons might be learned here, but I sort-of doubt it...

Also laughing at our ops guys who are now absolutely helpless and have people phoning them. I warned about this on our risk call and it was seen as low probability low risk. I hope it's down for a few hours :)

Sure, "New Outlook" on Windows is still entirely broken (including stuff like, oh, re-ordering mailboxes), but on Mac it's actually quite usable, and even allows you to switch off the IMAP proxying.

I want to believe this is the cause. It'd be especially funny because businesses were already anticipating the productivity loss when GTA6 comes out, but now we have actual productivity loss because of this outage (and it's affecting people who were never going to play the game anyway, lol)

If they roll it out evenly the week prior, then today (1/7th) is just 75 PB. To get that done in a single day would require a sustained 6.9 Tbps.

I'm sure this is all handled by a CDN at some level... but again it's not nothing.

Sure GTA6 may be a slight uptick in the amount of data being downloaded like any new game being released, but I would wager that the amount of data they are normally pushing out for downloads just on a regular basis (wether new purchases, redownloading games, game pass, etc) is already quite high.

451 4.3.2 Temporary server error. Please try again later ATTR2 [MI3PEPF00008486.eurprd02.prod.outlook.com 2026-11-12Txxxxxxxxxxxxxx]

  421 4.3.2 The maximum number of concurrent connections per resource forest has exceeded a limit, closing transmission channel. xyzxyzxyz.outlook.com. PRX8 [Name=InboundProxyTrackingAgent][AGT=IPT][MxId=xyzxyzxyz] [xyzxyzxyz.ausprd01.prod.outlook.com 2026-11-13T01:08:15.373Z xyzxyzxyz]

Sure, the program on your desktop computer is fine, but the Outlook app on your phone routes everything through MSFT's servers, so you may be impacted. Outlook.com is running EWS, so it would be down as well.

And finally, if you're talking to a layperson, they're not going to know what Exchange is. They just know they opened Outlook and it isn't working.

When your outsourced system to Microsoft goes down every few months your boss can’t yell at you

Outsourcing is about shifting blame elsewhere

From the individual's perspective Microsoft might seem like the better choice if they can achieve higher availability, but the societal cost is so much greater when the one basket you've put all your eggs in breaks even if everyone holding their own egg on average results in more broken eggs per year.

> I dislike Microsoft as much as the next guy, but many businesses impacted by these outages are completely innocent and don't deserve these consequences.

At no point is "keeping people alive" mentioned. Businesses use email for everything. Businesses usually don't care about where their email server is hosted. They just want to send emails. They shouldn't have to care either. I don't put blame on a business that happened to choose Outlook, one of the largest email providers, to host their email.

Deeply irresponsible.

> They shouldn't have to care either.

According to what realistic principle?

I'd rather Microsoft host it for me than have to maintain a team of people with Microsoft Exchange skills running either on a cloud environment (similar issues) or on my own infrastructure.

I for one don't miss the days of Exchange going offline or having issues and being the person having to try figure it out.

Microsoft has much better reliability than MOST companies.

The Status page is not working well, but I managed to subscribe to their RSS feed and it has been sending out updates periodically. I'll post some of them here.

9:30 PM UTC: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.

9:16 PM UTC: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've also moved traffic for some services off the affected infrastructure to further reduce load, with additional traffic moves in progress.

8:59 PM UTC: We've completed redistributing connection traffic across our worldwide infrastructure, and we're moving additional traffic off the affected infrastructure to further reduce load. We're also restarting affected servers to return them to service. In parallel, we're investigating whether the elevated traffic reflects a genuine increase in requests or a shift in how traffic is being routed across our infrastructure.

I am happy to no longer use PHP for anything.

I've used PHP in two previous jobs and certainly the stuff that I was in control of had it set to 0 because we made sure we didn't leak memory. Other departments... I'm sure at least one had the limit set to something non-zero.

It was the only thing running on that box so it wasn't a big deal to reboot.

If it "fixes" your problem, then you didn't fix you problem you identified it. It's another symptom of a bloated industry that can't or won't innovate, and I think it's more "won't" than "can't."

For example, we still can't patch running Windows, and most of the information technology professionals I know are still convinced it's impossible. Billions upon billions of dollars in time have been lost, probably enough man hours to send men to mars several times over.

Yet, you can just buy a version of Windows Server that already does it. It's not only possible, they already have the code. They just don't care enough to integrated it into the product most people use.

looks at the Boeing 787

I would argue a "just reboot it" culture leads to more problems -- engineers should be empowered to diagnose problems before they become catastrophic. In the world of software we even have mechanisms for extracting data from failed systems to root cause them later, and yet most people do not take that step before "just reboot[ing] it".

[1]: AirAsia 8501, Lauda Air 004, and SmartLynx Estonia 9001 come to mind.

A physicist, an engineer, and a computer programmer are driving down a steep mountain pass when the car’s brakes fail.

The car speeds out of control around sharp turns, nearly flying off the edge before they miraculously manage to skid to a stop at the bottom.

Shaken up, they discuss what to do next:

• The physicist says: "We need to calculate the friction coefficients in the brake pads and model the temperature rise to figure out why they failed."

• The engineer says: "I have a few tools in the trunk. I’ll take a look underneath and see if I can fix the mechanical failure."

• The programmer says: "Why don't we get back in and see if it's reproducible?"

It was a small fleet, so we had a pretty good idea if the same problem kept popping up so we could look more into that system.

A few fault codes were well known enough that if you called with that one the answer was to shut down and find a hotel, you aren't going anywhere and no reboot will take care of it. (bad detectors in the engines for about 10 years until a redesign)

we "bounce" the servers when on calls with customers.

We absolutely should have scheduled reboots during off hours. Lessons learned.

You know a lot of talented people who work there also post here, and it's not fair to insinuate that they aren't talented because some vastly complex system somewhere in the world had a bad day.

Stuff happens, things break. That's why lots of nines cost a lot of money. It doesn't automatically mean that anyone is incompetent.

lots of nines means not having significant downtime. we passed 99.99% hours ago, and we're on the way to 99.9%

Almost certainly not the fault of individual people, it's a management failure of course

I agree there are many talented people at Microsoft, but I don’t see the professionalism in this incident.

Isn't that comment saying the direct opposite? If it's a cultural problem, its precisely Satya's fault for not fixing it; that's the executives' job.

and at the top is an indian

and thus, the indian failure mode commences.

4

They transformed Windows from a useful OS to a banal surveillance and advertising platform.

Between breakfast, 1-hour lunch, 15-min ice cream break, 30-mins chill, 30-mins foosball games I think she worked 4 hours max.

At that point she caught some flak since it was timed with the layoffs. And I get that is not necessarily representative of the MS culture…

… but talk about timing and poor choices!

They should have established procedures for large-scale events, and they must include a coordinator whose entire job is to manage the external and internal communication.

Actual tech write-ups can come later.

In retrospect, the cause and cure are obvious to everyone.

Most of the time we get demonized by resume-driven developers job-hopping every few years. Today we get to laugh at their expense. These problems were entirely foreseeable and avoidable.

What a great phrase!, can I use that in conversation?

If we could see the memory bits and the CPU internals, I think it would be good candidate, particularly branching and branch prediction.

A brick is very complex. A bearing is complex. A bolt and a nut and a washer are complex. An integrated circuit is complex. We build stuff out of that. Like buildings that don't fall when you sneeze on them.

In software we use duct tape - at best. But that's not just Microsoft, that's pretty much everyone. With maybe few exceptions.

Plenty of internal accounts from previous employees highlight this.

The problem is amateur hour is the entire industry. Nobody specs anything, nobody tests overload conditions, nobody thinks failure modes. The stuff that every engineer does when building anything else other than software doesn't happen. The orgs are bloated. Too many teams. Too many silos. Incentives are stacked the wrong way. People are motivated to not think about systems but stay in their lane. Responsibilities are split in weird ways.

Yes. Big systems are hard. But we can build, manage and understand big systems with the right approach and people. Even one computer is complicated and many people get stuck trying to debug a piece of software running on one machine. At the end of the day it's not magic. All of humanity is big systems but when I have a power failure the power company is generally able to understand why and resolve it and the water in my tap doesn't stop flowing and the cell towers still work (generators, who would have thunk).

A friend who is really good had to leave Microsoft because of how AI was pushed. Before that he was telling me some other stories that make the culture seem pretty not great. Heard similar stories from other people. It used to be a great place filled with great people until at some point it's just another big company.

My bet is we'll eventually get there. Planes kill people when they fall, software used to not be as dangerous, but it's been a while since it became critical infrastructure.

Given that people on reddit claimed that in the entirety of New Zealand no real estate transactions can be settled, causing cascading failures [1]...

I think this might actually be the incident pushing governments to designate large online services such as MS 365, Gmail et al as critical infrastructure and force companies to massive, yet-unseen levels of transparency and regulation.

The EU is already itching to act ever since the Trump admin forced US Big Tech to terminate service to the ICC judges, and this here was a square own goal for Microsoft.

[1] https://www.reddit.com/r/sysadmin/comments/1xus565/comment/1...

That's not to excuse incompetence or whatever, just to note that the benefit of software is precisely how easy it is to change, which will always lead to more problems.

I remember a power outage in 2003 that knocked out power for 55 million people and took over a week to restore in many places.

Note that buggy software was a major contributing factor! The software was designed to, and would have, detected a deteriorating system an hour before anything broke, if not for a certain bug which caused the whole monitoring system to display stale data and then crash, including the redundant backup. This wasn't the only problem by far, but it was one layer of swiss cheese.

Until the cost is mentioned. People love free or nearly free software.

Also, power grids are a poor example. There's been multiple instances of absolutely catastrophic failures so hoity toity capital E Engineers also make mistakes.

https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou...

https://en.wikipedia.org/wiki/Northeast_blackout_of_2003

This is the inevitable consequence.

When I look across the scale of all the failures that have occurred in big tech over the past few years, my self-hosted solutions have been cheaper and more reliable. They are simple. They are small. They are easy to fix and diagnose. Because they're not trying to solve all the world's problems.

Managers and marketers are the ones asking for all those useless features. And god forbid they run out features to ask for, because then they start asking for architecture v2.0 projects.

In the name of efficiency and profit, we have stripped away all the safeguards that should prevent cascading failures after the first domino falls. Anything that doesn't return profit now, today, every day, must be purged. Because what if that domino never falls? Then you wasted your money! Think how much you can save if you don't need to make backups! (And every dollar saved is a dollar in my pocket!) Etc.

And torben-friis is right: the only solution to this, demonstrably, is regulation. Require multiple vendors for every major component. Require staffing sufficient to cover a third the company being on vacation. Require resilient systems, that can take a single disaster and keep on trucking, because they are the backbone of our economy.

It's really not intuitive, and at least for me, the need to admit that we couldn't just comp-sci our way out of some conditions was painful.

Site was taken offline, war rooms ensued. 48 hours later, an Ops team from Redmond was flown out. Another three days and they determined it was several legacy servers were going down and the load balancing servers were getting hit so hard, they also failed. It was a cascading effect that killed everything.

CIO was pissed. The company got absolutely lit up on every social media channel for their ineptitude. The CIO vowed it would never happen again, they were upgrading the entire network and infrastructure to avoid it happening again. Fast forward six months to the next major release. Same thing, same panic, same cascading failures because servers were collapsing under the traffic to the website. Another war room, another Ops team flies out. Two days later they figured out it was some buggy code that was causing it.

This after 8 rounds of load balancing testing to the tune of 10X the amount of traffic the site sees on these major release dates without so much as a hiccup.

I left two months later, but the overall feeling was the same - total amateur hour from Microsoft after our company was gifted major discounts to go all in with their hardware & network gear. The company found out the hard way it wasn't a smart decision.

The scary part was this some 20 years ago - seems like not much has changed considering how frequently Github is down now too.

They still don't know why this "core process" is maxing out the processor for no reason.

https://isolveproblems.substack.com/p/how-microsoft-vaporize...

https://news.ycombinator.com/item?id=47616242

I'm sure it's more complicated than that.... but from my POV, which is extremely limited and informed by what others online tell me.... that this is amateur hour

What would it take for it to be professional hour?

You can't do much to work around nuclear war or a major asteroid strike, but keeping important things working under normal operating conditions should not be a high bar.

Perhaps more accurate in our world.

The issue now is fixing this so status of what happens is needed and not how it happened in the first place.

The cause will ans should only be published when the issue is fixed and reviewed.

I feel for those stuck in there. The outage going on for this long has got to be fucking brutal.

From the 11:06:31 PM UTC update:

Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We've also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.

We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.

For reasons that I cannot fathom, Microsoft wants to act as the relay to my self-hosted e-mail account, if I add the account in Outlook.

I can conjecture some reasons. Datamining would be high on the list, since MS can read your mail and thus market to you more effectively. Governments might also be happier if MS can provide them with one-stop access (like Prism) to otherwise self-hosted e-mail accounts proxied through a central server.

BUT, more likely, it's some sort of "check in" or "dial home" feature that isn't working. Or plausibly, maybe they count "local user checked email" and their analytics endpoint is down.

Regardless, it's definitely surprising that a traditional desktop application would be blocking when only polling from local/trusted data sources.

It is normally probably more reliable; except when their server is down.

play stupid games win stupid prizes.

maybe its the support desk burnout talking, but if you rely on MS apps and services you're gonna have a bad time.

It's just funny that Outlook breaks this badly.

I'm being an ass in saying this, because I'm aware human motivation is a beautiful kaleidoscope of variables, but I would have guessed at the venn diagram of 'those willing to compromise and use a Microsoft product' is mutually exclusive with that of the 'I'm hardcore enough to self-host email'.

I will say, however, that Outlook used to be one of the better desktop email programs, so I'll grant some legacy lenience.

I use Gmail's web interface for my Gmail account (which, I'll admit, makes my little rant above more than a little hypocritical)

And in my self hosted email setup I use RoundCube (web interface).

Na, that would simply be the kaleidoscope part you mentioned, wouldn't it? For what it's worth, I found the combination just as peculiar as you did.

Almost as good as mutt.

Unfortunately, being someone who speaks up nowadays has been portrayed as being an asshole.

How dare you!?

In a corporate environment, I don't think this would be that unusual. But yes, it's hard to imagine someone would do that in a personal setup.

Biggest data heist and most obvious US intelligence operation in human history. I can't fathom why AVs don't treat "Outlook New" as info stealer malware.

I’m pretty certain because my company’s phishing-test emails skip my filters (I assume admins have a “skip filters” toggle). Which, hilariously, makes it even easier to ensure there’s a place that contains only phishing test emails (filter all not on allow-list of domains to one place, filter all on that list to another, ta-da, the main inbox now contains only phishing test emails)

I assume they do the same thing with non-MS email if you use their client, rather than maintain two filter implementations.

All around, yeah, wow, super gross.

In other words, a day that ends in "y".

My company was recently acquired by a company that uses Office, so I'm using Microsoft products for the first time in 15 years. It's absolutely APPALLING how bad they are. I assumed they got fairly usable in the last 15 years, but it seems they've gone backwards.

For all my gripes about GSuite, ... I don't know how businesses can justify using software this bad.

O365 is Microsoft Office shoehorned into running in a web browser.

I have a teams app with Outlook integration and received a bunch of 502s to smba.trafficmanager.net and a my logs are riddled with "RestError Temporary connection issue when contacting service. Please wait before retrying request"

I will say this, having apps deployed on Slack and Teams, Slack rarely has issues (an exponential backoff is enough to simply never worry about my Slack integration). With Teams I have mysterious failures to debug at least 2x a year - and they're often not very transparent about them (often my first instinct is to search Twitter and find people complaining about the same thing).

Outlook just popped an authentication dialog on me, too! That's more than it has been doing since 1300 Eastern.

I wonder if MSFT's corporate email is hosted outside the environment in a more "on-prem" style environment (i..e dedicated Exchange installs for MSFT's use). I can't imagine anybody inside MSFT would speak-up about it right now, but it would be interesting to know if they're fighting this thing w/o being able to use email themselves.

Everyone has outages but this is a biggie.

https://redmondmag.com/articles/2018/09/11/microsoft-explain...

I think it does _too much_ and the performance is _bad_ but it’s rock-solid for video and audio calls - more than even FaceTime or analog calls.

Using Teams after being used to Slack is like going from VSCode to Eclipse. I’m sure this thing is powerful, but it looks weird, it feels weird, it moves like January molasses, and it has a million features I’m completely uninterested in. I’m sure some people get value from all the baroque. I am not one of them.

(I’d rather be using IRC and Emacs but the world has other ideas.)

Truly, we are in dark times.

What else can you expect from Microsoft? Satya Nadella, the CEO of Microsoft is from India and as expected has outsourced so much to India.

> "Milken said earlier that if a US company replaced the US-born CEO with a CEO born in India, he buys the stock.

> The first thing they start doing is cutting US jobs and offshoring or automating them."

https://timesofindia.indiatimes.com/world/us/if-a-company-re...

That sounds bad

> [Home Affairs Minister Tony Burke] says the government has activated a digital resilience agreement with Microsoft today to ensure Australia gets priority:

> "That agreement has in fact been activated today to make sure that we get appropriate prioritisation of Australian interests in this across everything.

It's a little pathetic-seeming to plead for special treatment from Microsoft. I get Australia is an important strategic alliance* for the US, but unless these resiliency agreements are rare... what meaningful prioritisation of "Australian interests" can we expect from MS?

*a useful staging ground to field subs into the South Pacific and Indian Ocean

We use Azure at work and it’s wild how much stuff is just manual labor behind the scenes on their platform.

Seems like it’s always on fire somewhere, ripe for burnout and MBOs aren’t aligned with reality from what I hear.

I remember their customer-side push for semantically merging Office and some other suites into something-or-other like 365 or Copilot, creating enormous confusion. I have difficulty believing MSFT distinguishes between handmade and machine-learning tooling in that type of environment.

The amount of times we've had to reach out to an experienced person who had transferred to another team/org about processes in our team was ridiculous.

AWS is no different. Everything is on fire all the time, but customers usually don't notice.

Above those layers is a ton of jank, e.g. Elastic Beanstalk. But my understanding is the fundamentals on AWS - compute, storage, networking - are pretty much rock-solid.

Disclosure: I work at AWS, at the core EC2 dataplane, no less. We monitor our production environment very aggressively with very tight SLAs, and our technical leadership is competent as well.

(Although I like the idea of Management BuyOut, too.)

If the docs are correct I'm sure they'll depreciate the offering and rename it to something else and leave you on an island either way.

to what extent, though? over the course of my working with azure, ive been disappointed constantly, by just about every service i've used. things that should work a particular way don't, and when you try to investigate you end up in a maze of out-of-date and/or totally wrong documentation. i've started a project before, followed their precise "quick start!" guide to the letter, and find it totally non-functional and wrong.

things that DO work end getting updated (often due to financial incentives) and break, or shuddered entirely. some values that you might want to change for certain services are hard-coded (which, to their credit, you can verify from their public github repos) for ostensibly no reason. a lot of their documentation is internal only, and the public facing stuff feels perennially out-of-season.

(opinion) i feel like fundamentally microsoft has embraced ai development so wholly because they've not been able to describe their services in documentation very well, and thus rely heavily on the ai's ability to digest what scraps of truth remain and the various source code to be effective at delivering output

But this is certainly exactly what I'd expect after indiscriminately laying off half your workers, then immediately offering a buyout, to make sure any experienced people left can find the door.

If this incident, or the difficulty they're having with remediation, are somehow not caused by any of that, then it's at least a preview of what things are going to be like.

What changed is the productivity per employee: 2021-2026: +60% 2024-2026: +38%

To be honest, I also assumed your take on this at first. But after a bit of research I found quite a different reality!

The total populace did not change, because it was replaced by new "talent".

Btw does everyone see how insanely crappy Windows 11 is compared to MacOS

The irony is that I can finally get out of the Microsoft ecosystem thanks to Microsoft. The only thing that was keeping me there before is Visual Studio (which runs reliably only on Windows). But with the advent of LLMs and Visual Studio Code (which is cross platform), I haven't touched Visual Studio in months, and doubt I ever will again. LLMs should also ease the transition into a Linux ecosystem. RTFM? Nah ATFLLM.

One has to spend more on third party tools for getting MacOS to feature parity with a windows than a Win Pro license...

Linux is fine on the other hand, but you compared to a joke of an operating system.

This is just an anecdote, but I have the exact opposite experience. I'm a C++ games developer working on Windows 11 every day and other than the micro frustrations with stupid UI, the system just works. I know I can rely on it day to day. But I was also given a MacBook last year for our iOS ports and I don't understand how anyone considers it a serious operating system for work - it's full of bugs to a point where I started using AI to work around some of them. I thought maybe MacOS 27 would fix some of them and sure it did - but then it introduced a whole bunch of new ones. As a developer it's probably the worst OS you could pick - if you hate windows for whatever reason go straight to Linux, at least you have actual control over your OS compared to MacOS where basic things like spotlight are broken and there's no way to fix them other than add to the thousands of complaints on apple forums.

Honestly, for personal machines the M series is wonderful hardware, and I have young kids so no time to tune Linux (I ran it at home for around a decade or so).

But yeah, the interface has many, many bugs.

My "favourite" (new) bug is that sometimes you can't move a window from the laptop screen to another screen. It goes away if you full screen the app, but it's bizarre to me that it made it to production.

Then due to typical clueless management and lack of taste and complete buffoonery, they just fucked it all up. I'm not complaining because I think everyone is better off, but it's just amazing to have witnessed that even when they luck out on a bet, they manage to fuck it up. Copilot started as a coding assistant, and before it could breath they destroyed the brand by shoving the name into literally everything without thinking or taste. Ignored the coding part, because that's lame, and instead focused on half assed Windows or Office apps and integrations that no one will ever use. Once claude code dominated the scene, they realized the major fuckup and tried to catch up, but it's too little too late.

Again, not complaining really.

I actually know quite a few (non-tech) people that love to use Copilot in those apps. So I'm not sure it's as big of a fuck-up as you think it is (or as I certainly used to think it is).

On profitable products though; sure as a general concept, but the rate of return, opportunity cost and market competition factor into the profitability equation too. You could go waste your time building a desktop app for a chatgpt clone and make some "profit" in the same time your competitor threw similar resources at making a claude code and that makes 100x your profit. That would mean you actually lost despite the profit. The opportunity cost isn't only in building the wrong thing, but also in the human resources you allocated to the task that deprived other areas. I see no world where the 6 dozen copilots we got from Microsoft have any good ROI beyond diluting a brand and making them more of a laughing stock.

We need to rewire ourselves to think about things that are beneficial to society as a whole. Not "profitable." Profit just means more money going from the many to the few. That's what's gotten us into this awful situation.

So fortunately, there's not really an incompatibility there.

Windows does always cater to backwards companility and making IT departments happy, so that explains some of the lagging technology and systems. I can't believe NTFS is still the default file systems. That's generations behind Apple's APFS.

The stock has popped, and that's what Wall Street cares about, but I could see a world where this chronic underinvestment and mismanagement basically implodes the entire company in the coming years.

New Xbox to cost $1200-$1800.

Gamepass limited to 15 hours a month for Ultimate.

New Xbox cost is not confirmed.

It works perfectly on a fast home connection over ethernet, mind blowing how responsive it is and load times are non existent.

But as far as I can tell most people subscribe to game pass anyway? Why buy the hardware at all when I can just cloud game in the same subscription. .

Here, Series S is £400-£430 for 512GB or £500-£520 for 1TB.

Xbox Series X is £600-£620 for digital 1TB or £50 more for the disc version. That's much more than I paid for mine a year after launch (£450 I think), but luckily still nowhere near 1700 Euros.

Indeed you’re seeing the consequences of their past decade+ of management

A quick google search turned up these global sales numbers: PS3 87.40 million, PS4 117.20 million, PS5 96.44 million. Xbox 360 85.73 million, Xbox One 57.96 million, Xbox Series X/S 35.68 million.

It seems to be back in stock now, but at a cost of about 1000 Euros for a Series/S I'm not really feeling enticed even if GTA 6 is around the corner. They probably really shot themselves in the foot by not having units out a year ago and now fewer will buy due to pricing.

I think Sony is shooting themselves in the foot with the digital only consoles too - unless they maintain a very attractive set of exclusives at all times, people will just choose whichever digital subscription is best or cheapest for them instead of something tied to a particular console.

They need to sell consoles to get people to become subscribers.

Nothing to do with today’s outage though.

Xbox Game Pass was an attractive offer for consumers when MS was getting people onboard and the price was so heavily subsidized that it could never have been a sustainable model. Once they tried to start bumping the price into something profitable people immediately said "okay well that's clearly not worth it, I'll just cancel."

They attempted to make it $30/month last year before backtracking. At $360/year, even factoring in the new $70 price of AAA new releases and ignoring you can wait a few months to buy them cheaper, you had to be buying 5 new release games per year at launch just to get close to breaking even with the subscription price. 6 before you save money. So...

> It's a really attractive offering for people who don't play games all that much (that is, more than 2 hours a day)

No, it's absolutely not worth it for these people. Even at the current prices they backtracked to ($280/year if you want to play new releases on day 1), if you're a person who doesn't game that much, it's cheaper to just buy a few new games per year. If you're not a frequent gamer, why would you of all people need access to a massive library of games?

I personally have a backlog of 45 games I want to play written down in a note. And those are just the ones I wrote down. Most of these games you can get for peanuts. I just played through Fallout New Vegas for my first time. It regularly goes on sale for $0.99, or $5 if you want all the DLC. I got 90 hours of playtime out of that.

The idea any casual gamers should be paying for a subscription service to a Netflix-style game library is nuts to me. These aren't movies and tv shows you'll be flying through every evening, they're big games that can take weeks to play even if you're putting in a few hours per day.

I see what you did there.

Fabric is a weird one, as it's sortof a GUI over spark, but also PowerBI and lots of weird services.

It feels like a "data plaform" for people who've never used one before and only have the vaguest idea of what it means.

That being said, Spark is pretty good and we're running a lot of workflows on it (and probably building more on top, I suspect that I'll be much angrier about it in approximately 6 months or so).

Ui is clunky, they are terribly slow.

The only reason companies use Entra is it appears cheaper but eventually you pay as much as competitors. Okta is a way better IAM solution if you need one. Although it’s very pricey.

Yeah, and additionally if all your counterparties are on it, you can externalise your internal tools onto them relatively easily.

This mostly doesn't work, but it's a beautiful dream for executives.

I agree. All the execs need to do is tell Copilot to "fix this so that it never happens again. Make no mistakes" and run it on bypass permissions. Tada!

Maybe they can use azure hosted openai models? Or are they down as well?

Just was a perfect opportunity to showcase astra and fable and other models

If this continues the company will see a lot of customers leaving. And they were already in a bind since Europe is finally realizing that its dependence on American tech could be used against them.

They're not corrupt (in a lot of cases), mostly just ignorant. Like, they don't understand the tech, and microsoft is safe, they're already using it so handing over large swathes of their infra to them seems reasonable.

MS have wonderful sales people, and this is why they do well with governments and large enterprises.

There's also a co-ordination problem, in that all of your counterparties (like your citizens, for instance) are familiar with MS and therefore it costs a bunch locally to move, for little short-term reward.

This is actually a place where (good) regulation could derisk a lot of these changes, but you'd need a committed group of politicians who can convince the civil servants to keep at it. It'll happen, but slowly and in starts over the next 5-15 years.

If they transfer this to sovereign providers it would cost Big Tech serious money. So they're quietly lobbying to slow down the speed at which European nations are moving away from them.

Yeah, this is definitely really important to them (not important enough for them to comply with the laws, but whatevs).

This absolutely predates AI.

I think this is the logical conclusion of using copilot to write your stuff for the last two years. Nobody knows what it does and more and more bugs are being solved by just pointing the AI back to it again. Combine with the fact that so many people have been laid off and replaced with copilot I'm not surprised that the few who remain aren't quite sure how to solve it

After that they will tackle email servers.

So.. > Be the last standing SRE at Micro$slop > Your mentors have all left, taking the exit payout, now your days are the best b/c you answer to nobody and can hide in your mountain of markdown. > You knew each of the 7 different products named CoPilot are used to monitor your performance. So you set them all up talking to eachother, burning tokens and API limits. > You are team top performer now, you get all the important decisions now > You walk in the office this morning, CISA is sitting at your desk, with CVE-2026-124194 on your whiteboard > 'You have a patch for this right, China is actively exploiting our Defense primes..'? > Oh no, I haven't landed a patch in prod since my mentor left (because he actually did most of the work) > Oh well, must be close enough, LGTM, CISA seems upset so better just yeet this to prod... > ...

inb4 this is more accurate than anyone wants it to be.

I mean, either roll back you update or there is a malicious actor involved.

https://en.wikipedia.org/wiki/SQL_Slammer

https://en.wikipedia.org/wiki/Conficker

Current status: As part of our investigation into the sustained traffic management imbalances, we’re reviewing traffic flows and affected infrastructure to identify opportunities to optimize routing. This includes verifying the reasons for service failures so we can develop specific actions to improve stability.

We will continue to share more information as soon as we have it, and for now we can confirm that there is no indication this is the result of malicious activity. We’ll continue to apply learnings towards remediation actions as we confirm effectiveness mitigation actions.

Issues that cause outages are complex and can cascade across multiple areas around the world.

Designing for resiliency doesn't mean you won't ever have a major outage.

One of them has two servers on AWS behind a load balancer. That's the very least of resilience.

Another one has one server in a datacenter running their primary service on a single VM. That server had issues with RAM chips a couple of years ago that brought it down a few times. However they are not crazy. They made the choice to keep their data at home, bought reportedly good hardware, accepted there there might be consequences. They have other less beefy servers but they did not load balance their service on VMs on different servers.

I think that we can agree that this is no resilience. Is it by design as in "I want my service not to be reliable"? Of course not. Is it lack of quality in the design? Of course it is. But does it matter from the point of view of the users of the service?

At work I designed DNS to survive the loss of a city, so again 100% uptime. The riskiest part is deploying changes, which is why the deployment doesn't run in parallel and confirms state before deployment.

Resilience isn't hard when you aren't trying to serve a billion customers. The problem is we are addicted to a "winner takes all" economy, which leads to fragility. If Tesco breaks, I can go to Sainsburys, or Aldi, or Lidl, or Iceland, or get in the car and go to Asda or Morrisons. That's diversity.

But in IT the goal is all those follow the same gartner reports and all use the "best supplier" for their backend infrastructure, and if that supplier breaks then they can't restock any supermarkets.

That's a terrible risk and needs to be stopped, not encouraged. But the individual incentives are for the CTO of each supermarket to choose what the consultants tell them. If they don't they are accountable for the choice of not using things like "industry standard".

And that's anathema to companies nowadays.

In a sane world the CTOs choosing this would be held accountable. The massive global economic fragility which is created by the centralisation of society is a massive risk. Diversity is important, we see it time and time again how optimisation causes panic, from crowdstrike to microsoft. When your outsourced provider is "too big to fail", you have a problem that government needs to deal with.

Best of luck to those oncall.

It's 7PM PST now and I cannot imagine how fucking on fire things must be for it to not only still be down, but basically with no real communication from what i've seen (like yes there's updates, but not the "root cause found, triaging".)

In an era where everyone's vibe coding Excel clones, this is NOT the time to be fucking up your KEY moat on a catastrophic level.

Yeah, I'll stick to Notepad and my local TODO.txt file...

> Current status: We’ve monitored the status of the changes applied to improve serviceability and confirmed that change is not taking as expected. We’re identifying alternate workstreams that can be applied to increase capacity in the face of sustained traffic management imbalances.

I take it you've never seen a Windows file copy dialog :)

Anyone hoarding weights will become a dissident.

>Previous status: We’ve monitored the status of the changes applied to improve serviceability and confirmed that change is not taking as expected. We’re identifying alternate workstreams that can be applied to increase capacity in the face of sustained traffic management imbalances.

>Current status: As part of our investigation into the sustained traffic management imbalances, we’re reviewing traffic flows and affected infrastructure to identify opportunities to optimize routing. This includes verifying the reasons for service failures so we can develop specific actions to improve stability.

What a dumpster fire

https://www.youtube.com/watch?v=t62ZRncv2d4

Throttle the hell out of everything, then slowly ramp traffic back up.

I'm confident they've already tried this exact thing. This is cloud 101 in 2026.

> Current status: As part of our ongoing efforts to advance the recoverability of our frontend servers, we’re evaluating traffic controls that can be applied to increase capacity and stability. This includes traffic optimization actions such as traffic throttling, protocol adjustments, and failover routing intended to improve stability in active regions. In parallel, were expanding internal testing to track activity within a contained environment to improve traffic monitoring and issue detection capabilities.

That...kinda reads like they're just trying it? I'm assuming there's something being missed or it's a "fuck it try it again" but I do wonder if some story is going to come out of people in the decision making seats who don't know what they're doing.

10 hours later, Outlook still down

Without anything else to go off of it points to it certainly being a capacity, traffic, routing issue.

Or maybe ai broke containment and it’s doing this for the lulz.

Atrocious sentence.

For hours they’ve been saying some variation of “we changed something but we can’t tell if it worked”. Can’t imagine what the vibe is like on this incident bridge.

One can only hope. But I am not so sure. No one seems to want to employ a local sysadmin these days.

Think of yourself as a blue-collar contractor providing services to white-collar people afraid of being taken advantage of. Your competitive advantage over Geek Squad is being the local IT person who knows the names of everyone in a five-person real estate or law office.

A few more outages though.. maybe?

I can tell you somewhat anecdotally that people who usually forget about this stuff are starting to notice.

The 2011 Blackberry outage burried RIM..... companies ran away from them faster than a greyhound with its arse on fire.

Note that the infra for M365, Azure etc is different. They would be running on servers hosted in MSFT DCs but have completely different CI/CD pipelines. M365 services have their own deployment pipelines and it takes on the order of weeks to deploy your changes to production.

Considering the fact that Outlook is down, clearly email doesn’t seem to be working. So email uses something called Microsoft Exchange Server, there used to be an on-premises version of this but MS stopped supporting it to run the cloud version hereby called M365. Email is one big “macro-service”, if you’re running Windows Server, look at the process called “EdgeTransport.exe” this is responsible for sending/receiving email on Windows Servers. There are probably multiple 100s of services supported by 100s of teams that are run while email is routed from sender to receiver via MSFT Exchange servers. My recollection is that even if one of these services goes down or introduces latency in the pipeline, it caused a major incident. When I was working in M365 Security in 2024, this Exchange Server pipeline was called “legacy” because it’s one big .NET Framework service. I don’t know if they upgraded to .NET core but developing on this was as HN comments say, very janky.

The reason Teams doesn’t seem to be down is that Teams services are completely different and are written using .NET Core and have their own separate release pipeline. It was much easier to interact with Teams services. I actually worked closely with the team that was responsible for ingesting Teams messages to the customer’s data store. Their code base was much more modern and easy to understand. Exchange Server not so much.

I don’t know if what I said is accurate but know that Exchange server code was like a fossil that didn’t change as much, over the ~5 years I worked on it from 2019-2024.

Story time over.

Is this what it feels like to be old?

The Exchange code base frankly was a behemoth. No one had a full understanding of how it worked E2E. I am just sharing how I understood it. I was only a lowly L63 btw.

Oh, the memories. That login screen with the changing hieroglyphics [1] is forever burned into my mind.

[1] https://web.archive.org/web/20080220070921/http://lotusnotes...

lmao... but is it actually random or is it a hash of your password to make sure you typed it correctly?

I remember people phoning to say "I"ve sent you an email, press the manual refresh button to get it faster".

up to this day I ask if companies use Lotus and those who remember just shake their heads

If you want immediate response use IRC or actually phone someone.

Now it’s more like being subscribed and listening for events.

And in the 1990s, people were just adopting email, and their existing context was mail that arrived daily. Other than IRC, nobody had ever experienced any instant instantaneous form of communication other than phone calls, so nobody really felt like they were missing anything if they didn’t get notified about an email for 15 minutes.

Today by contrast, I feel like 3/4 of the time I go into my email client it’s to get some login code sent by some stupid website, so from this perspective 15 minutes sounds like absolute torture.

So when I read about Exchange Server to me it still feels quite modern.

The latter is probably the case for most enterprises.

Getting releases deployed into the testing environments, much less production, requires coordination, testing, reports and sign-offs.

Of course this can normally be expedited or bypassed for incident management and critical fixes.

"Microsoft Teams: Most of the core functionality including joining calls, meetings, group chats, and channel messages are not impacted. Scenarios that depend on Exchange Online, including Calendar related failures with scheduling, invites, viewing, and meeting edits. Creation of Live or Virtual events fail. Failures with Copilot responses for outlook, calendar, or M365 files, read history, or adding to new chats. Navigating to App chats, adding new contacts or syncing contacts in settings fail. Standard channels are working fine but users may experience the inability to create or delete private or shared channels."

After many enterprise customers refused to migrate Exchange Server SE exists since 2025 and will be supported until at least 2035.

Though given how the quality of their updates are going one could still say that it's "not supported", but at least they are charging money for it!

I must live in a bubble... the last time I heard of anyone using Outlook was when Scott McNealy was calling it LookOut! in the 90s.

In a ~35 year tech career I have never been exposed to, nor used Windows nor Outlook and have never met anyone who does. Who are these companies that still use it?

You have probably mostly worked for tech companies where business employees were not the primary workforce. When you think about it most other business generally have everyone needing a word, excel or PowerPoint. It bundles cleanly with the whole package.

I'm assuming your "surprise" isn't genuine.

Yes, then you probably at least work in a bubble.

In my experience, it's more common than GSuite in most companies outside of startups and big tech and is ubiquitous at schools, among many, many other entities.

The CTO had to pull a bunch of strings to let me run Linux.

I work at a fortune 500 company and they got me, 3 hours after I started working , the basic info about the obvious the internet is already screaming at you.

Since we are responsible for a large web application I tried to reach out to people through VIVA (yes, we only make use of that) and we managed only to reach about 2-3% of the user base. Information-wise it's like a blackout. For living in 2026 this is completely unacceptable.

That itself wouldn't be a problem - email is very decentralised.

The problem is that we thought that putting like 2 companies in charge of most email communication is a good idea.

Your email is down? Cool now you can’t log into anything.

If we were mainly using Teams or whatever other "better than email" service and that had an outage, we'd actually be in the dark!

As the resident complainer about Microsoft lock-in, I was honestly quite surprised by how well we could work around this. Don't get me wrong, if it was Entra that fell over, we'd be thoroughly fucked, but Exchange wasn't all that bad.

On-premises Exchange is still supported and licensed. It's a subscription instead of a perpetual license.

> I don’t know if what I said is accurate but know that Exchange server code was like a fossil that didn’t change as much, over the ~5 years I worked on it from 2019-2024.

Exchange is a mature product (albeit not without its flaws). Its era of change between its 2007 and 2013 versions (inclusive). The 2016, 2019, and now Subscription Edition codebases were more of an evolution and has much less churn.

What this leaves is a comparison of whether on-premise exchange servers have been compromised or down compared to the cloud.

Exchange, in particular, is sort of "feature complete" to me. I'm happy w/ what it does and how it does it for the most part.

It's been a long time since I admin'd an Exchange server, but this still surprises me. But maybe it shouldn't surprise me since even Office is migrating completely online.

Pick. Up. The. God. Damed. Phone. And do your job.

My knowledge of this isnt great, but I remember being given a briefing on some of this years ago.

surly MS can afford flagship models.

    Outlook not so good

Gotta love the PR damagecontrol speak for global outage.

So my bet is if this is in any way AI related - perhaps the reasoning should go the other way around and is not mere dumbness of otherwise great company everyone’s buying like crazy for years now. Even my diehard Apple fans friends own MS stock.

Occam's razor.

Abscence of evidence vs evidence of absence notwithstanding, it’s probably best not to overly speculate.

   At   this   point  I'm  surprised  no   one's   registered
   [[ https://IsMicrosoftStillDown.Com/ ]] and points it to a
   static page that just says "yes."

I did a 20-year stint in IT security (quit when my company was bought by a capital fond that wanted us to still do all the things but preferred not to pay wages.)

I still don't feel qualified to guess what happened here - since I know next to nothing about this incident.

But darn it, it's not like we have anything else to do...

So here's my wild (emphasis) guess.

a) Lots of organisations have an "allow any" hole in their firewall for ike+ipsec, usually for reasons like "that protocol seems complicated, firewalling it would be complicated", more or less.

b) There used to be botnets that could pacify routers with modified GRE datagrams, specifically the "Mirai" botnet, operating from hijacked webcams (doesn't exist anymore afaik).

c) New botnets have surely emerged, and improved on the capabilities, maybe building on the publicized source code.

d) Just to give a random example, the "Proxynvader" botnet hijacks TV sticks; assessments vary but some experts say a quarter million, some say up to 8 million devices, each capable of delivering 0.1 Gbps into a volumetric attack.

I think it wouldn't be far-fetched to imagine one of those botnets grew the capability to spoof fx. ipsec datagrams, and that someone out there decided it would be fun if Microsoft's server parks started spending more time on fragment reassembly and decryption and less time doing whatever Microsoft thinks those servers should be doing. And then began blasting them with traffic on fx. 4500/udp.

(So in this example, traffic is routed around the normal ingress path through load balancers, tls offload, etc. for those servers, hitting them directly on a public IP via a permissive firewall rule instead.)

I think my odds of being correct are about 1% :-), since little is publicly known yet (and I personally know much much less than that.)

I enjoy reading the other theories here; some very interesting ones!

Haven't given much thought to motivation, could be anything. (I guess having an impact on the equity price means there's theoretically a profit to be made though.)

(... tldr: "ddos maybe?" ...)