It's impacting all businesses running MS hosted Outlook, that would also include companies with 100,000 of employees. It's a global issues not just one area.

Well Microsoft made everyone move to managed exchange/outlook right? I was told you can’t self host anymore. How did they not design for regional resiliency?

What makes you think they haven't?

Issues that cause outages are complex and can cascade across multiple areas around the world.

Designing for resiliency doesn't mean you won't ever have a major outage.

Well, no one designs to not be resilient. Quality varies in practice.

My customers are small.

One of them has two servers on AWS behind a load balancer. That's the very least of resilience.

Another one has one server in a datacenter running their primary service on a single VM. That server had issues with RAM chips a couple of years ago that brought it down a few times. However they are not crazy. They made the choice to keep their data at home, bought reportedly good hardware, accepted there there might be consequences. They have other less beefy servers but they did not load balance their service on VMs on different servers.

I think that we can agree that this is no resilience. Is it by design as in "I want my service not to be reliable"? Of course not. Is it lack of quality in the design? Of course it is. But does it matter from the point of view of the users of the service?

[delayed]

You design to the needs. I have resilience in my home DNS so I have a 100% availability bounded on the home (if the home is burnt down then I'll lose DNS)

At work I designed DNS to survive the loss of a city, so again 100% uptime. The riskiest part is deploying changes, which is why the deployment doesn't run in parallel and confirms state before deployment.

Resilience isn't hard when you aren't trying to serve a billion customers. The problem is we are addicted to a "winner takes all" economy, which leads to fragility. If Tesco breaks, I can go to Sainsburys, or Aldi, or Lidl, or Iceland, or get in the car and go to Asda or Morrisons. That's diversity.

But in IT the goal is all those follow the same gartner reports and all use the "best supplier" for their backend infrastructure, and if that supplier breaks then they can't restock any supermarkets.

That's a terrible risk and needs to be stopped, not encouraged. But the individual incentives are for the CTO of each supermarket to choose what the consultants tell them. If they don't they are accountable for the choice of not using things like "industry standard".

While it's probably true that no one specifically seeks to remove resiliency for its own sake, basically the entirety of our corporate world has, over the past several decades, been relentlessly redesigning everything in ways that remove resiliency. Because resiliency is almost inherently antithetical to efficiency. At the most fundamental level, if you want to make sure you have a backup of your data, that means you need at least 2x the amount of storage. And, of course, if you want to make sure you can operate even if a critical employee gets sick/is hit by a bus/goes on vacation, you need to hire extra employees.

And that's anathema to companies nowadays.

You can self-host, but it's a monthly subscription instead of a perpetual license.

Is there any proof that "self hosted" is still working, wouldn't surprise me if at least part of it relies on centralised services.

In a sane world the CTOs choosing this would be held accountable. The massive global economic fragility which is created by the centralisation of society is a massive risk. Diversity is important, we see it time and time again how optimisation causes panic, from crowdstrike to microsoft. When your outsourced provider is "too big to fail", you have a problem that government needs to deal with.

I haven't had occasion to spin-up Exchange SE yet, but I've read plenty of descriptions of other admins running it. I have no reason to believe Exchange itself relies on any centralized features. "New Outlook" has been alleged to use centralized servers even when connected to an on-prem Exchange SE install, and given that it's "modern" MSFT software that wouldn't surprise me, but anybody stupid enough to use "new Outlook" gets what they deserve.

[delayed]

US-East-1 is at least something you can kinda plan for / around. This isn’t.

Yet HackerNews soldiers on.

Can't take down simplicity this easy.

Yeah, with a brief outage, the concern is that you don’t have access. At this point, I’m worried that someone else does.

They're still trying to find the root cause apparently. The status updates I've seen (the page seems dodgy, probably millions of people hammering it every millisecond) don't sound very optimistic as of yet.

Just tell copilot to rollback the last 24h of changes, ez-peazy /s

Really should ask: how badly designed is O365 if their AI can’t solve it?

It's the Great Rebootening. That's what they're trying apparently. Roll back and reboot and see if everything works.

Soon to be O364 :-(

Best of luck to those oncall.

Copilot is down, they might need to ask a Chinese model.

Windows Recall: You Can See Everything

Hey, at least they fixed the status page. A few hours earlier, it was loading a 16MB JavaScript file over 2 minutes or something. It's much faster now, though I haven't checked if that file is smaller, or just the pipe got better.

It's still loading a 16MB JavaScript file.

But it's loading a 16MB JavaScript file much faster! Improvement (not in the direction I like, but still)!

When I first saw this I commented it must be bad, but that was 4 hours ago after a whole day of "welp i guess i'm not emailing anyone and meetings are pushed.".

It's 7PM PST now and I cannot imagine how fucking on fire things must be for it to not only still be down, but basically with no real communication from what i've seen (like yes there's updates, but not the "root cause found, triaging".)

In an era where everyone's vibe coding Excel clones, this is NOT the time to be fucking up your KEY moat on a catastrophic level.

I mean, they still have zero idea what the root cause is 12 hours later.

That's fucking terrifying if true. Guess i'm going to have to watch the tracker as well. I'll live but another "history in the making" moment

Not sure why the execs and VPs aren't asking their AI agents to triage. They've been implying they don't need engineers anymore, isn't this the best time to prove themselves right?

The last email received by my partner’s work inbox was a cheerful update from their CTO or whatever about the wonderful progress they had made by moving all employees to work primarily in OneDrive. I kind of wonder if he’s just run away since then.

Imagine how frequently this will happen when old employees take the buyout option. Coupled that with the constant layoffs, the nect decade looks promising for MSFT

Almost 8AM in central europe and outlook is still not working.

Midnight in the west coast now and my 20k+ institution is still communicating via SMS ...

Wrath of the Europeans that still host their own mail servers.