News for hackers. Stuff that matters.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by DanHulton on Sunday November 15, 2026 @03:21PM (#50510305)
Decades of “root cause analysis” poisoning have ruined our ability to think about safety and reliability holistically.

There isn’t one root cause here or one person or entity at fault. Many things went wrong and bad choices were made by many people and entities to lead us to this point, and they ought to be all regarded seriously and addressed rigorously.

- Whomever at the hotel that approved this as the sole (or even primary) alert system is at fault. - The lack of training and experience this hotel employee had when in a position to make such a dangerous choice was at fault. - Ditto the alarm company - everyone involved in designing, approving, and implementing this system is at fault, as well as whatever process ensured that nobody had the experience or safety to be able to speak up about this dangerous design is at fault. - The regulatory system that allowed this design is at fault. - The health and safety inspection system that didn’t catch this risk is at fault.

I’m sure there’s more; I’m just spitballing here. But the point is that there’s no one person or thing to blame here, and no one fix. This should kick off a wave of personal and professional soul-searching and improvements.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by Alive-in-2025 on Sunday November 15, 2026 @03:51PM (#50510607)
I don't think we ruined our ability to think about things, this was a failure to build a reasonable system. It's just basic planning and trying to consider possible faults. If the only way you will be informed of a potentially deadly situation, a sensor reading out of range - that affects human health - is it okay if it is only via email???

What if the person who reads the email was gone that day? What if they forget to read it? What if the email system is down? What if it was lost? This is just careless and bad design.

At the very least you need a backup notification, like say a warning siren or noise at the pool? Probably you don't want to do that casually because "it could scare people".

It's not some tricky thing where you had multiple ways to detect it, giving some redundancy. Same thing if it was a text message only or a phone call only.

This is a poor safety design with a single failure mode on the notification.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by techiferous on Sunday November 15, 2026 @04:44PM (#50511146)
Agreed. This is just poor system design.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by tosti on Monday November 16, 2026 @02:34AM (#50515041)
Back when the Germans did everything in-house, they managed to gas a _LOT_ more people.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by YokoZar on Sunday November 15, 2026 @05:04PM (#50511335)
Just because the alert signal is an email at one point in its journey from sensor to human doesn't mean you're relying on someone using a computer to see it. Automated paging systems can read email too.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by Alive-in-2025 on Sunday November 15, 2026 @05:28PM (#50511530)
Yes, but we have to remember email is not guaranteed to be delivered, it's not reliable. You cannot in general tell if someone even looked at it (people try to build systems on top to provide these things, but it's not universal).

Also, I have a personal domain, some sites decided its not real enough and I can't get email from them.

Email is just not reliable. It could be part of a combined system of other things that are reliable.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by hdgvhicv on Sunday November 15, 2026 @05:42PM (#50511650)
You’ve got a chain of multiple single points of failure. That’s unacceptable.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by YokoZar on Sunday November 15, 2026 @07:00PM (#50512271)
Of course. But it's bad for that reason, not because of a presumed requirement for a person looking at an outlook inbox like OP implied.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by shiandow on Sunday November 15, 2026 @05:42PM (#50511649)
You don't even need something exotic for this to fail. A simple power failure would disable this system and most backups.

We've been through this with smoke detectors, it's not a new thing.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by trollbridge on Sunday November 15, 2026 @05:56PM (#50511784)
The system should be designed to fail-safe during a power loss. If it can't, then it needs to have appropriate battery (and possibly generator) backup.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by hluska on Sunday November 15, 2026 @10:54PM (#50513819)
Your analogies aren’t equivalent because physical security and health and safety are quite hard.

The power failure would presumably shut down the pumps and end in the pool being closed. Then pools should have a power outage protocol they’d follow before opening the pool again. Checking chemistry would be high up on the list.

Smoke detectors need backups because a fire can cause a power outage before there’s enough smoke to trigger the smoke detector.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by II2II on Sunday November 15, 2026 @05:48PM (#50511712)
> At the very least you need a backup notification, like say a warning siren or noise at the pool? Probably you don't want to do that casually because "it could scare people".

More properly said, you may cause panic. People genuinely don't know how to respond when they hear an alarm in an unfamiliar place.

And that's if you're lucky. I once had to deal with people who did not want to evacuate a building when a fire alarm went off. And when I got them out the door I had trouble convincing them to leave the building's deck. Some people seem to think that it is okay to ignore a fire alarm, even a probable false alarm, in a wood structure with a wood deck.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by db48x on Sunday November 15, 2026 @07:24PM (#50512464)
It wouldn’t need to be that loud. Unlike a fire that becomes dangerous in mere minutes, this type of accident takes days to unfold. Most automatic pool systems will have a light or a beeping noise to let the operator of a failure. The email is just a convenience. The hotel staff should have been checking the system directly every morning to see if there were errors before they opened the pool for the day.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by wongarsu on Monday November 16, 2026 @03:42AM (#50515418)
If you only need employees to react you can replace the rotating red light and klaxon with a smaller flashing white light that plays a pleasant but slightly annoying "ding" sound every five seconds

It just has to be annoying enough that even the most junior employee would radio in their supervisor to get it to stop

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by trollbridge on Sunday November 15, 2026 @05:56PM (#50511782)
We have a general design principle that we assume important emails will not be delivered, will not be read, and will not be actioned on.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by helterskelter2 on Monday November 16, 2026 @01:18AM (#50514610)
I generally ignore emails until I get bugged about something 2-3 times or the subject line is all caps.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by gregglain on Sunday November 15, 2026 @04:13PM (#50510814)
First this is so sad and preventable. I would be interested in the "follow the money" on this.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by perching_aix on Sunday November 15, 2026 @04:22PM (#50510909)
[delayed]
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by ponector on Sunday November 15, 2026 @05:23PM (#50511481)
In real life shared responsibility means no one is responsible for it.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by perching_aix on Sunday November 15, 2026 @05:34PM (#50511600)
Hah, fun that you bring that up. Just had quite the struggle articulating to management a few weeks ago that "shared ownership" runs contrary to the very idea of ownership, which I guess is basically the same topic just with different words. Either we take shares ownership seriously and actually organize and form consensus on things, or it's just bullshit. Naturally, did not really lead anywhere. So you definitely have a point there.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by ponector on Monday November 16, 2026 @03:20AM (#50515277)
Shared ownership is perfect: if project succeeded you can claim you've contributed heavily to the success. If project failed - you did everything toward success, but other team members failed.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by delta_p_delta_x on Sunday November 15, 2026 @06:21PM (#50511958)
> Decades of “root cause analysis” poisoning have ruined our ability to think about safety and reliability holistically.

This is not the case in aviation. Even in the most egregiously obvious single-cause failures, there's always an analysis of issues in the wider system that allowed that single cause to become the single cause in the first place.

I agree with your points, though. In this case, Microsoft absolutely takes shared responsibility. It is only a matter of time until a similar outage in some other wider networked system with a so-called 'five/seven/nine nines uptime guarantee' goes out for a couple of days, and takes with it some critical life-sustaining system, and people die.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by thephyber on Sunday November 15, 2026 @08:21PM (#50512886)
This seems wrong (obviously I don't have any more info than you).

Ford doesn't build all of the modules of their cars. They design specs and outsource manufacturing. But legally speaking, Ford is responsible for design, testing, and support / recalls. It's their responsibility to contract in the terms that both protect their customers' safety and Ford's liability due to spec / contract violations.

When your company is responsible for the health of staff and guests while dealing with deadly gasses, it's your company's responsibility to ensure that all of your vendors detail how their failure modes fail safe.

This is the same exact logic that has been migrating software companies to detail their risk by demanding their vendors all supply a "Software Bill of Materials" (which is very difficult in practice, but is necessary for the downstream users to protect themselves from upstream risk).

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by notatoad on Sunday November 15, 2026 @10:01PM (#50513510)
the point of root cause analysis isn't to find the one person or entity at fault. the point is to determine what you can fix, so this doesn't happen again next time.

anything that was "at fault" that you can't do anything about doesn't matter. and you can't do anything about a microsoft outage, all you can do is design around it.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by wallst07 on Monday November 16, 2026 @01:23AM (#50514636)
Run the 5-Whys on this, start with:

"Why did people get gas poisoning"

https://en.wikipedia.org/wiki/Five_whys

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by ponector on Monday November 16, 2026 @03:22AM (#50515283)
Because the are too much alive. AI robots cannot get gas poisoning. Case closed!
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by thaumasiotes on Monday November 16, 2026 @04:13AM (#50515583)
"Five Whys" is never going to mean anything as a methodology. It works if competent people use it and it fails if incompetent people use it. But pretty much everything works if you're free to assume that everyone involved is competent.

The point of a process is to constrain incompetent people into doing something that will work despite their incompetence. Five Whys is entirely subjective and can never constrain anyone; it's just something that some people think sounds deep.