Most of the time we get demonized by resume-driven developers job-hopping every few years. Today we get to laugh at their expense. These problems were entirely foreseeable and avoidable.
What a great phrase!, can I use that in conversation?
If we could see the memory bits and the CPU internals, I think it would be good candidate, particularly branching and branch prediction.
A brick is very complex. A bearing is complex. A bolt and a nut and a washer are complex. An integrated circuit is complex. We build stuff out of that. Like buildings that don't fall when you sneeze on them.
In software we use duct tape - at best. But that's not just Microsoft, that's pretty much everyone. With maybe few exceptions.
Plenty of internal accounts from previous employees highlight this.
The problem is amateur hour is the entire industry. Nobody specs anything, nobody tests overload conditions, nobody thinks failure modes. The stuff that every engineer does when building anything else other than software doesn't happen. The orgs are bloated. Too many teams. Too many silos. Incentives are stacked the wrong way. People are motivated to not think about systems but stay in their lane. Responsibilities are split in weird ways.
Yes. Big systems are hard. But we can build, manage and understand big systems with the right approach and people. Even one computer is complicated and many people get stuck trying to debug a piece of software running on one machine. At the end of the day it's not magic. All of humanity is big systems but when I have a power failure the power company is generally able to understand why and resolve it and the water in my tap doesn't stop flowing and the cell towers still work (generators, who would have thunk).
A friend who is really good had to leave Microsoft because of how AI was pushed. Before that he was telling me some other stories that make the culture seem pretty not great. Heard similar stories from other people. It used to be a great place filled with great people until at some point it's just another big company.
My bet is we'll eventually get there. Planes kill people when they fall, software used to not be as dangerous, but it's been a while since it became critical infrastructure.
Given that people on reddit claimed that in the entirety of New Zealand no real estate transactions can be settled, causing cascading failures [1]...
I think this might actually be the incident pushing governments to designate large online services such as MS 365, Gmail et al as critical infrastructure and force companies to massive, yet-unseen levels of transparency and regulation.
The EU is already itching to act ever since the Trump admin forced US Big Tech to terminate service to the ICC judges, and this here was a square own goal for Microsoft.
[1] https://www.reddit.com/r/sysadmin/comments/1xus565/comment/1...
That's not to excuse incompetence or whatever, just to note that the benefit of software is precisely how easy it is to change, which will always lead to more problems.
I remember a power outage in 2003 that knocked out power for 55 million people and took over a week to restore in many places.
Note that buggy software was a major contributing factor! The software was designed to, and would have, detected a deteriorating system an hour before anything broke, if not for a certain bug which caused the whole monitoring system to display stale data and then crash, including the redundant backup. This wasn't the only problem by far, but it was one layer of swiss cheese.
Until the cost is mentioned. People love free or nearly free software.
Also, power grids are a poor example. There's been multiple instances of absolutely catastrophic failures so hoity toity capital E Engineers also make mistakes.
https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou...
This is the inevitable consequence.
When I look across the scale of all the failures that have occurred in big tech over the past few years, my self-hosted solutions have been cheaper and more reliable. They are simple. They are small. They are easy to fix and diagnose. Because they're not trying to solve all the world's problems.
Managers and marketers are the ones asking for all those useless features. And god forbid they run out features to ask for, because then they start asking for architecture v2.0 projects.
In the name of efficiency and profit, we have stripped away all the safeguards that should prevent cascading failures after the first domino falls. Anything that doesn't return profit now, today, every day, must be purged. Because what if that domino never falls? Then you wasted your money! Think how much you can save if you don't need to make backups! (And every dollar saved is a dollar in my pocket!) Etc.
And torben-friis is right: the only solution to this, demonstrably, is regulation. Require multiple vendors for every major component. Require staffing sufficient to cover a third the company being on vacation. Require resilient systems, that can take a single disaster and keep on trucking, because they are the backbone of our economy.
It's really not intuitive, and at least for me, the need to admit that we couldn't just comp-sci our way out of some conditions was painful.
In retrospect, the cause and cure are obvious to everyone.