I am happy to no longer use PHP for anything.
I've used PHP in two previous jobs and certainly the stuff that I was in control of had it set to 0 because we made sure we didn't leak memory. Other departments... I'm sure at least one had the limit set to something non-zero.
It was the only thing running on that box so it wasn't a big deal to reboot.
If it "fixes" your problem, then you didn't fix you problem you identified it. It's another symptom of a bloated industry that can't or won't innovate, and I think it's more "won't" than "can't."
For example, we still can't patch running Windows, and most of the information technology professionals I know are still convinced it's impossible. Billions upon billions of dollars in time have been lost, probably enough man hours to send men to mars several times over.
Yet, you can just buy a version of Windows Server that already does it. It's not only possible, they already have the code. They just don't care enough to integrated it into the product most people use.
looks at the Boeing 787
I would argue a "just reboot it" culture leads to more problems -- engineers should be empowered to diagnose problems before they become catastrophic. In the world of software we even have mechanisms for extracting data from failed systems to root cause them later, and yet most people do not take that step before "just reboot[ing] it".
[1]: AirAsia 8501, Lauda Air 004, and SmartLynx Estonia 9001 come to mind.
A physicist, an engineer, and a computer programmer are driving down a steep mountain pass when the car’s brakes fail.
The car speeds out of control around sharp turns, nearly flying off the edge before they miraculously manage to skid to a stop at the bottom.
Shaken up, they discuss what to do next:
• The physicist says: "We need to calculate the friction coefficients in the brake pads and model the temperature rise to figure out why they failed."
• The engineer says: "I have a few tools in the trunk. I’ll take a look underneath and see if I can fix the mechanical failure."
• The programmer says: "Why don't we get back in and see if it's reproducible?"
It was a small fleet, so we had a pretty good idea if the same problem kept popping up so we could look more into that system.
A few fault codes were well known enough that if you called with that one the answer was to shut down and find a hotel, you aren't going anywhere and no reboot will take care of it. (bad detectors in the engines for about 10 years until a redesign)
we "bounce" the servers when on calls with customers.
We absolutely should have scheduled reboots during off hours. Lessons learned.
You know a lot of talented people who work there also post here, and it's not fair to insinuate that they aren't talented because some vastly complex system somewhere in the world had a bad day.
Stuff happens, things break. That's why lots of nines cost a lot of money. It doesn't automatically mean that anyone is incompetent.
lots of nines means not having significant downtime. we passed 99.99% hours ago, and we're on the way to 99.9%
Almost certainly not the fault of individual people, it's a management failure of course
I agree there are many talented people at Microsoft, but I don’t see the professionalism in this incident.
Isn't that comment saying the direct opposite? If it's a cultural problem, its precisely Satya's fault for not fixing it; that's the executives' job.
and at the top is an indian
and thus, the indian failure mode commences.
4
They transformed Windows from a useful OS to a banal surveillance and advertising platform.
Between breakfast, 1-hour lunch, 15-min ice cream break, 30-mins chill, 30-mins foosball games I think she worked 4 hours max.
At that point she caught some flak since it was timed with the layoffs. And I get that is not necessarily representative of the MS culture…
… but talk about timing and poor choices!
They should have established procedures for large-scale events, and they must include a coordinator whose entire job is to manage the external and internal communication.
Actual tech write-ups can come later.
In retrospect, the cause and cure are obvious to everyone.
Most of the time we get demonized by resume-driven developers job-hopping every few years. Today we get to laugh at their expense. These problems were entirely foreseeable and avoidable.
What a great phrase!, can I use that in conversation?
If we could see the memory bits and the CPU internals, I think it would be good candidate, particularly branching and branch prediction.
A brick is very complex. A bearing is complex. A bolt and a nut and a washer are complex. An integrated circuit is complex. We build stuff out of that. Like buildings that don't fall when you sneeze on them.
In software we use duct tape - at best. But that's not just Microsoft, that's pretty much everyone. With maybe few exceptions.
Plenty of internal accounts from previous employees highlight this.
The problem is amateur hour is the entire industry. Nobody specs anything, nobody tests overload conditions, nobody thinks failure modes. The stuff that every engineer does when building anything else other than software doesn't happen. The orgs are bloated. Too many teams. Too many silos. Incentives are stacked the wrong way. People are motivated to not think about systems but stay in their lane. Responsibilities are split in weird ways.
Yes. Big systems are hard. But we can build, manage and understand big systems with the right approach and people. Even one computer is complicated and many people get stuck trying to debug a piece of software running on one machine. At the end of the day it's not magic. All of humanity is big systems but when I have a power failure the power company is generally able to understand why and resolve it and the water in my tap doesn't stop flowing and the cell towers still work (generators, who would have thunk).
A friend who is really good had to leave Microsoft because of how AI was pushed. Before that he was telling me some other stories that make the culture seem pretty not great. Heard similar stories from other people. It used to be a great place filled with great people until at some point it's just another big company.
My bet is we'll eventually get there. Planes kill people when they fall, software used to not be as dangerous, but it's been a while since it became critical infrastructure.
Given that people on reddit claimed that in the entirety of New Zealand no real estate transactions can be settled, causing cascading failures [1]...
I think this might actually be the incident pushing governments to designate large online services such as MS 365, Gmail et al as critical infrastructure and force companies to massive, yet-unseen levels of transparency and regulation.
The EU is already itching to act ever since the Trump admin forced US Big Tech to terminate service to the ICC judges, and this here was a square own goal for Microsoft.
[1] https://www.reddit.com/r/sysadmin/comments/1xus565/comment/1...
That's not to excuse incompetence or whatever, just to note that the benefit of software is precisely how easy it is to change, which will always lead to more problems.
I remember a power outage in 2003 that knocked out power for 55 million people and took over a week to restore in many places.
Note that buggy software was a major contributing factor! The software was designed to, and would have, detected a deteriorating system an hour before anything broke, if not for a certain bug which caused the whole monitoring system to display stale data and then crash, including the redundant backup. This wasn't the only problem by far, but it was one layer of swiss cheese.
Until the cost is mentioned. People love free or nearly free software.
Also, power grids are a poor example. There's been multiple instances of absolutely catastrophic failures so hoity toity capital E Engineers also make mistakes.
https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou...
This is the inevitable consequence.
When I look across the scale of all the failures that have occurred in big tech over the past few years, my self-hosted solutions have been cheaper and more reliable. They are simple. They are small. They are easy to fix and diagnose. Because they're not trying to solve all the world's problems.
Managers and marketers are the ones asking for all those useless features. And god forbid they run out features to ask for, because then they start asking for architecture v2.0 projects.
In the name of efficiency and profit, we have stripped away all the safeguards that should prevent cascading failures after the first domino falls. Anything that doesn't return profit now, today, every day, must be purged. Because what if that domino never falls? Then you wasted your money! Think how much you can save if you don't need to make backups! (And every dollar saved is a dollar in my pocket!) Etc.
And torben-friis is right: the only solution to this, demonstrably, is regulation. Require multiple vendors for every major component. Require staffing sufficient to cover a third the company being on vacation. Require resilient systems, that can take a single disaster and keep on trucking, because they are the backbone of our economy.
It's really not intuitive, and at least for me, the need to admit that we couldn't just comp-sci our way out of some conditions was painful.
Site was taken offline, war rooms ensued. 48 hours later, an Ops team from Redmond was flown out. Another three days and they determined it was several legacy servers were going down and the load balancing servers were getting hit so hard, they also failed. It was a cascading effect that killed everything.
CIO was pissed. The company got absolutely lit up on every social media channel for their ineptitude. The CIO vowed it would never happen again, they were upgrading the entire network and infrastructure to avoid it happening again. Fast forward six months to the next major release. Same thing, same panic, same cascading failures because servers were collapsing under the traffic to the website. Another war room, another Ops team flies out. Two days later they figured out it was some buggy code that was causing it.
This after 8 rounds of load balancing testing to the tune of 10X the amount of traffic the site sees on these major release dates without so much as a hiccup.
I left two months later, but the overall feeling was the same - total amateur hour from Microsoft after our company was gifted major discounts to go all in with their hardware & network gear. The company found out the hard way it wasn't a smart decision.
The scary part was this some 20 years ago - seems like not much has changed considering how frequently Github is down now too.
They still don't know why this "core process" is maxing out the processor for no reason.
https://isolveproblems.substack.com/p/how-microsoft-vaporize...
I'm sure it's more complicated than that.... but from my POV, which is extremely limited and informed by what others online tell me.... that this is amateur hour
What would it take for it to be professional hour?
You can't do much to work around nuclear war or a major asteroid strike, but keeping important things working under normal operating conditions should not be a high bar.
Perhaps more accurate in our world.
The issue now is fixing this so status of what happens is needed and not how it happened in the first place.
The cause will ans should only be published when the issue is fixed and reviewed.
I feel for those stuck in there. The outage going on for this long has got to be fucking brutal.
From the 11:06:31 PM UTC update:
Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We've also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.
We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.
The Status page is not working well, but I managed to subscribe to their RSS feed and it has been sending out updates periodically. I'll post some of them here.
9:30 PM UTC: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.
9:16 PM UTC: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've also moved traffic for some services off the affected infrastructure to further reduce load, with additional traffic moves in progress.
8:59 PM UTC: We've completed redistributing connection traffic across our worldwide infrastructure, and we're moving additional traffic off the affected infrastructure to further reduce load. We're also restarting affected servers to return them to service. In parallel, we're investigating whether the elevated traffic reflects a genuine increase in requests or a shift in how traffic is being routed across our infrastructure.