News for hackers. Stuff that matters.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by N19PEDL2 on Sunday November 15, 2026 @02:28PM (#50509854)
This is 100% fault of the hotel, not of Microsoft. It’s unacceptable that the communications regarding such a dangerous thing totally rely on email, without any backup or double check.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by macNchz on Sunday November 15, 2026 @02:34PM (#50509909)
Seems like if the system can send an email it should be able to, like, illuminate a red light somewhere or something, if what it’s doing can create an actual safety risk.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by freitasm on Sunday November 15, 2026 @02:38PM (#50509940)
Came here to say this. A push notification to an app or Slack/Discord. Or even a light.

I know the S in IoT means safety/security, but there must be things that can be done for urgent notification.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by croes on Sunday November 15, 2026 @03:03PM (#50510155)
I doubt they have Slack or Discord and I bet there is nobody who constantly watches a light.

You know what they all have? An email account

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by temac on Sunday November 15, 2026 @05:12PM (#50511403)
> A push notification to an app or Slack/Discord

Wtf?

I must have missed the memo where Slack Discord or “apps“ became suitable for safety critical task. Must be what USA has tried to use just before grounding all planes.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by cbarnes99 on Sunday November 15, 2026 @02:39PM (#50509951)
It's the fault of the company that designed the automatic alarm. Not the hotel. But it is accurate to say that the cause of the issue was the outage.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by 01HNNWZ0MV43FF on Sunday November 15, 2026 @02:48PM (#50510031)
We can assign fault, or we can ask which levers we can pull to make it better:

1. Can there not just be a light / siren right next to the gas sensors? Easy to fix, but it requires hardware changes on-site.

2. Could Microsoft have not had an outage? Hard to fix.

3. Could they use some better notifications systems like idk self-hosted ntfy.sh with a warning if the server goes down? Yes but nobody wants to pay for that.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by croes on Sunday November 15, 2026 @03:09PM (#50510208)
A light has to be seen, a siren has be heard. I bet email has a better chance of alarming the right person.

And you have to check those lights and sirens that they still work.

Everybody knew email was down.

The real problem is nobody thought of the consequences regarding the alarm system

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by stackskipton on Sunday November 15, 2026 @03:24PM (#50510333)
Generally, for sensitive safety systems like this, Light and Siren are visible to those using the facility. So if problem developed, all patrons would be going "Umm, you know your sauna has flashing red light and siren going off?"

Yes, they still have to work but first off, both of those are extremely reliable systems, alarm circuits have extremely low risk of failure. Second, testing them takes 5 minutes a week/month, press the button, look for flashing light and listen to wailing noise.

It's inexcusable to think that email/slack or whatever is acceptable as anything other than notification system so people not in facility can be notified that alarms are going off.

EDIT: You are defending this setup and I'm not sure why. This seems email only setup seem indefensible.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by croes on Sunday November 15, 2026 @03:57PM (#50510670)
Because we don’t live in an ideal world.

Testing requires staff and you can be sure those parks are understaffed.

Best thing is to avoid them, next best thing is a cheap solution with low maintenance by the staff to notify as much people as possible.

Email wins.

Not a good solution but the best you will get for the price they are willing to pay

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by ssl-3 on Sunday November 15, 2026 @04:41PM (#50511114)
Things do break, so all alarms need periodic testing, whether they consist of email and/or something else.

Whether it breaks because a relay has corroded due to years of service in a chlorine-enriched environment, or because O365 is down, or because O365 tightened up on DMARC requirements: It needs tested.

And it's not a binary choice. This isn't the World Cup, the Super Bowl, or Highlander. There can be more than one.

The alarm system can wear a belt, and suspenders. It can send emails, and it can flash a bright light, and it can also sound an audible alarm.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by icehawk on Sunday November 15, 2026 @08:04PM (#50512776)
> A light has to be seen, a siren has be heard.

Curious: what do you think has to be done with an email?

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by sokols on Sunday November 15, 2026 @02:50PM (#50510050)
Well somebody has to sign off the solution, in this case the hotel.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by buran77 on Sunday November 15, 2026 @02:51PM (#50510060)
> It's the fault of the company that designed the automatic alarm. Not the hotel.

The hotel picked a monitoring solution which relies exclusively on email, or configured it like that. If anything they are absolutely both to blame but for different things. The company for designing a safety related system like this, the hotel for selecting an inadequate solution/configuration thus allowing their guests to be harmed.

> But it is accurate to say that the cause of the issue was the outage

The cause of the issue is the more immediate one - the configuration of the monitoring system. This avoids going up the causality chain to far - shitty coffee caused engineer to take down email system, and such.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by zardo on Sunday November 15, 2026 @02:57PM (#50510107)
Email going down is the proximate cause. A system configuration relying on a single unreliable communications route is further up the causal chain.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by buran77 on Sunday November 15, 2026 @05:56PM (#50511778)
Let's not forget that we're talking about people getting hurt in a hotel which has a responsibility towards their guests, that's the core topic here, yes? The most reasonable proximal cause for the people getting hurt is that there was too much chlorine in the water. Because the monitoring system didn't work properly. Because the email failed. Because someone deployed a wrong config on MS's cloud infra. And so on.

As I said, there's plenty of blame to go around as a matter of general principle. But for what happened here the hotel bears the responsibility towards their guests, not the monitoring system developers, not Microsoft.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by stackskipton on Sunday November 15, 2026 @03:31PM (#50510395)
Agreed, I'm not sure how it's acceptable to say "Email is exclusive option you can have" vs "We can email you when alarm is going off but light/siren will flash and it must be visible to those using the facility"
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by ithkuil on Sunday November 15, 2026 @04:09PM (#50510790)
It would be accurate to call it the proximate cause.

Using the unqualified "cause" is misleading click bait and in some places it could get you sued

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by spunker540 on Sunday November 15, 2026 @04:30PM (#50510996)
Isn’t the cause whatever sprung a leak in the first place? Ie a faulty pipe or valve?
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by ktm5j on Sunday November 15, 2026 @04:56PM (#50511259)
I feel like there are things we aren't being told here.. what kind of automatic chlorine dosing system both knows that there's too much chlorine and continues to add enough chlorine to the point of being harmful. Something is fishy here...
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by mmoll on Monday November 16, 2026 @01:55AM (#50514805)
The control loop probably used email for messaging.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by Dylan16807 on Sunday November 15, 2026 @07:29PM (#50512501)
It's not the alarm that was dumping in dangerous levels of chlorine. Microsoft is in the line of causes to the lack of warning, but not the actual poisonous conditions.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by xyst on Sunday November 15, 2026 @02:52PM (#50510067)
hindsight is 20/20.

I do agree with this sentiment. At the same time, it’s still unacceptable for a global service like MS365 to go down for extended periods of time. I wouldn’t be surprised if the hotel chain was sold on some MS guarantee that it would never become unavailable.

Although I would also believe a scenario where the hotel chain "IT” (aka the owners son or grandchild) just used default alert settings and didn’t bother to setup fallback alerting mechanisms.

In short, all parties suck here (MS and hotel)

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by Espressosaurus on Sunday November 15, 2026 @03:02PM (#50510148)
Hindsight is 20/20? Bullshit. Safety critical systems should not rely on transport layers that are guaranteed to fail. Email especially! Physical alarms are the way here. The email should be for convenience and to inform higher-ups.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by reaperducer on Sunday November 15, 2026 @03:08PM (#50510201)
Safety critical systems should not rely on transport layers that are guaranteed to fail.

Go invent a transport layer that will never fail, and make yourself rich.

Because right now, there is no such thing.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by Espressosaurus on Sunday November 15, 2026 @03:11PM (#50510230)
Everything can fail even down to individual capacitors on the board but that doesn’t make the failure probabilities or failure mode equivalent to something like email that is best effort only and can be actively blocked by things like spam filters that aren’t even under your control
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by Rygian on Sunday November 15, 2026 @03:23PM (#50510326)
"Loud noises and flash bright red lights" is an industry- and regulations-accepted transport layer for a life-or-death signal.

Email is not.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by reaperducer on Sunday November 15, 2026 @05:09PM (#50511374)
"Loud noises and flash bright red lights" is an industry- and regulations-accepted transport layer

And yet, still not "guaranteed," which was the criteria specified by the OP.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by cwillu on Sunday November 15, 2026 @06:04PM (#50511843)
Please quote where the OP said “guaranteed”; all I see is a claim that email is guaranteed to fail.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by cwillu on Sunday November 15, 2026 @03:49PM (#50510576)
Audible alarms, failsafe heartbeats and scheduled networking are not new technologies.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by reaperducer on Sunday November 15, 2026 @05:09PM (#50511377)
Audible alarms, failsafe heartbeats and scheduled networking are not new technologies.

And yet, still not "guaranteed," which was the criteria specified by the OP.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by sham1 on Monday November 16, 2026 @01:18AM (#50514605)
I know this is HN and all, but you're being way overly pedantic about the "guaranteed to fail" phrasing of the person you originally replied to.

Yes it's true, no system is guaranteed to function, but for Pete's sake, it's clear to everyone that having only an email-based alarm system for chlorine levels is the wrong thing to do just because "well there's no alarm system that is guaranteed to alert people, so let's rely only on email, tee-hee".

Any redundancy would have been better than this. People's lives were at risk, and here we are, debating the meaning of the word "guaranteed". I'm going to be frank here, are you an MSFT shareholder or tied to the hotel in question?

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by greggsy on Sunday November 15, 2026 @02:52PM (#50510069)
Exactly. Email is also a ‘zero guarantee’ service, and should never be relied upon for life or death solutions.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by mikeryan on Sunday November 15, 2026 @03:22PM (#50510324)
Not just that, in this case there was no way for the alert system to even know that the emails were delivered. I can’t speak to everyone but for my company emails were not bouncing, I just got them 12-24 hours late. At least SMS and the like know if the message reached the actual end recipients inbox.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by InsidiousWombat on Sunday November 15, 2026 @04:46PM (#50511167)
SMS is also zero guarantee service
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by _bernd on Sunday November 15, 2026 @05:47PM (#50511703)
Please chill out. Chlor Level in a swimming pool is now where life and death or as the other user here noted, the user of the pool should be able to notice that something wrong by them self. You still have think a little bit for your self to prevent self harm.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by ssl-3 on Sunday November 15, 2026 @06:21PM (#50511957)
15 people had respiratory issues. 4 were sent to a hospital. 1 was kept there.

> the user of the pool should be able to notice that something wrong by them self.

Clearly, their motivations for autonomous self-preservation were insufficient.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by _bernd on Monday November 16, 2026 @03:04AM (#50515204)
oh you sweet usofa boys. Just because the ambulance brings you to the hospital for monitoring doesn't make your condition life threatening nor gives you bankruptcy. I still hold the point. To much chlorine in the water should be noticable otherwise I would argue you shouldn't even be allowed or able to buy a ticket because your are unable to perform proper choises.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by ashleyn on Sunday November 15, 2026 @02:59PM (#50510121)
This seems like the kind of thing which should have an audible alarm mounted on the wall in the pool room.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by croes on Sunday November 15, 2026 @03:03PM (#50510161)
You want a panic? That’s how you get a panic
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by mastax on Sunday November 15, 2026 @03:07PM (#50510188)
How do you feel about fire alarms?
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by garyfirestorm on Sunday November 15, 2026 @03:26PM (#50510349)
You want a panic? That’s how you get a panic /s
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by fc417fc802 on Sunday November 15, 2026 @03:40PM (#50510475)
People just panic and do ridiculous things like evacuating the apartment building at 2 am in their pajamas in the rain, blocking the road. Meanwhile 9 times out of 10 it's a false alarm. We should instead email the fire department while the residents keep sleeping. /s
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by ssl-3 on Sunday November 15, 2026 @04:26PM (#50510962)
Good. When they're outside in their PJs standing in the rain and blocking the road like complete dullards, then: They're not in the building, and they are neither on fire nor breathing chlorine gas.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by Dylan16807 on Sunday November 15, 2026 @07:34PM (#50512543)
Yes, that is the point of the comment you responded to.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by ssl-3 on Sunday November 15, 2026 @09:19PM (#50513242)
[delayed]
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by Dylan16807 on Sunday November 15, 2026 @09:54PM (#50513461)
Agreeing is fine if you have something to add.

But a "Good." to that kind of sarcasm looks like a lack of understanding, and then you rephrased the joke as a non-joke with nothing extra. If that's the full intent, I'd say just upvote.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by ssl-3 on Sunday November 15, 2026 @10:41PM (#50513753)
I see.

So what's the value of the contribution of your own comments in this thread? If you have nothing to add, I'd say: Just move along.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by Dylan16807 on Monday November 16, 2026 @12:44AM (#50514438)
Letting you know.

It's not much value but if it leads to better outcomes in the future then that's a success.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by asgraham on Sunday November 15, 2026 @05:01PM (#50511306)
You’re being sarcastic but that’s a weirdly popular view.

I called 911 once because of two CO monitors going off in my top floor apartment of a mid-rise. Half a dozen engines responded including hazmat, everyone’s standing around outside the building. FD had to break down the door of someone who wasn’t home because there was no response. Somehow it got around that I had been the one to call me in. Some guy came up to me saying something like “some of us have jobs in the morning. I hope you’re happy, moron.”

FD found a leak in the basement furnace. Without the alarms, everyone in the building would’ve been dead by morning.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by croes on Sunday November 15, 2026 @03:45PM (#50510541)
Had a fire alarm on my last vacation.

People ignored it.

A message to the fire department would have been the better idea

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by RetroTechie on Sunday November 15, 2026 @04:54PM (#50511240)
Most o/t time they're just annoying. But if I'm in a building & hear one, I will ALWAYS check where it comes from, and whether further action (like evacuation) is needed.

It's easy & the 1 time you don't, can cost liveS.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by windowsrookie on Sunday November 15, 2026 @03:10PM (#50510214)
So what if people panic? People panicking because of an alarm is better than being poisoned...
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by s0rce on Sunday November 15, 2026 @03:12PM (#50510237)
Facilities I worked in that had chlorine gas had audible alarms if leaks are detected because everyone could die.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by skeledrew on Sunday November 15, 2026 @03:24PM (#50510334)
So it's better to have deaths rather than panic?
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by DougN7 on Sunday November 15, 2026 @03:32PM (#50510412)
Panic seems like exactly what you want in this situation.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by cowsandmilk on Sunday November 15, 2026 @03:55PM (#50510636)
They literally had to call the fire department and send 4 people to the hospital. That is what causes panic.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by shepherdjerred on Sunday November 15, 2026 @04:07PM (#50510764)
That's exactly why I deleted my team's metrics, dashboards, and alerts
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by ssl-3 on Sunday November 15, 2026 @04:24PM (#50510935)
[delayed]
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by _trampeltier on Sunday November 15, 2026 @04:26PM (#50510955)
And then the horn or the cable is broken when you need it. So you need a redundant system for important things.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by sparrish on Sunday November 15, 2026 @05:20PM (#50511454)
Then you test the horn and cable daily (maybe even hourly if it level could become toxic that quickly) to make sure they are working.

Similar to doing regular restores of your data backups. If you never test a restore, you don't have a backup.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by _trampeltier on Monday November 16, 2026 @12:49AM (#50514464)
Yes, things like cable loop test for such cases exist since ever. But what do you do if it fails? Send an email?
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by namibj on Monday November 16, 2026 @03:58AM (#50515509)
Use the hot spare cable loop to yell at people. No occupancy without two working ones; this means you want a third so you can do maintenance without having to evict occupants first.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by InterlooperX on Monday November 16, 2026 @03:29AM (#50515331)
That, and gauges telling you what's what and Dashboards mounted and so on and so forth.

I did an apprenticeship as an electrical engineer and have been in those catacombs underneath such water park type of things. There should be such devices present, and of course personnel to operate them.

I mean, can you imagine your boss' face when you tell em the reason your entire system is down is you didnt get the email from checkmk that something is wrong? Instead of, you know, checking the dashboard yourself periodically.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by DanHulton on Sunday November 15, 2026 @03:21PM (#50510305)
Decades of “root cause analysis” poisoning have ruined our ability to think about safety and reliability holistically.

There isn’t one root cause here or one person or entity at fault. Many things went wrong and bad choices were made by many people and entities to lead us to this point, and they ought to be all regarded seriously and addressed rigorously.

- Whomever at the hotel that approved this as the sole (or even primary) alert system is at fault. - The lack of training and experience this hotel employee had when in a position to make such a dangerous choice was at fault. - Ditto the alarm company - everyone involved in designing, approving, and implementing this system is at fault, as well as whatever process ensured that nobody had the experience or safety to be able to speak up about this dangerous design is at fault. - The regulatory system that allowed this design is at fault. - The health and safety inspection system that didn’t catch this risk is at fault.

I’m sure there’s more; I’m just spitballing here. But the point is that there’s no one person or thing to blame here, and no one fix. This should kick off a wave of personal and professional soul-searching and improvements.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by Alive-in-2025 on Sunday November 15, 2026 @03:51PM (#50510607)
I don't think we ruined our ability to think about things, this was a failure to build a reasonable system. It's just basic planning and trying to consider possible faults. If the only way you will be informed of a potentially deadly situation, a sensor reading out of range - that affects human health - is it okay if it is only via email???

What if the person who reads the email was gone that day? What if they forget to read it? What if the email system is down? What if it was lost? This is just careless and bad design.

At the very least you need a backup notification, like say a warning siren or noise at the pool? Probably you don't want to do that casually because "it could scare people".

It's not some tricky thing where you had multiple ways to detect it, giving some redundancy. Same thing if it was a text message only or a phone call only.

This is a poor safety design with a single failure mode on the notification.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by techiferous on Sunday November 15, 2026 @04:44PM (#50511146)
Agreed. This is just poor system design.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by tosti on Monday November 16, 2026 @02:34AM (#50515041)
Back when the Germans did everything in-house, they managed to gas a _LOT_ more people.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by YokoZar on Sunday November 15, 2026 @05:04PM (#50511335)
Just because the alert signal is an email at one point in its journey from sensor to human doesn't mean you're relying on someone using a computer to see it. Automated paging systems can read email too.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by Alive-in-2025 on Sunday November 15, 2026 @05:28PM (#50511530)
Yes, but we have to remember email is not guaranteed to be delivered, it's not reliable. You cannot in general tell if someone even looked at it (people try to build systems on top to provide these things, but it's not universal).

Also, I have a personal domain, some sites decided its not real enough and I can't get email from them.

Email is just not reliable. It could be part of a combined system of other things that are reliable.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by hdgvhicv on Sunday November 15, 2026 @05:42PM (#50511650)
You’ve got a chain of multiple single points of failure. That’s unacceptable.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by YokoZar on Sunday November 15, 2026 @07:00PM (#50512271)
Of course. But it's bad for that reason, not because of a presumed requirement for a person looking at an outlook inbox like OP implied.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by shiandow on Sunday November 15, 2026 @05:42PM (#50511649)
You don't even need something exotic for this to fail. A simple power failure would disable this system and most backups.

We've been through this with smoke detectors, it's not a new thing.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by trollbridge on Sunday November 15, 2026 @05:56PM (#50511784)
The system should be designed to fail-safe during a power loss. If it can't, then it needs to have appropriate battery (and possibly generator) backup.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by hluska on Sunday November 15, 2026 @10:54PM (#50513819)
Your analogies aren’t equivalent because physical security and health and safety are quite hard.

The power failure would presumably shut down the pumps and end in the pool being closed. Then pools should have a power outage protocol they’d follow before opening the pool again. Checking chemistry would be high up on the list.

Smoke detectors need backups because a fire can cause a power outage before there’s enough smoke to trigger the smoke detector.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by II2II on Sunday November 15, 2026 @05:48PM (#50511712)
> At the very least you need a backup notification, like say a warning siren or noise at the pool? Probably you don't want to do that casually because "it could scare people".

More properly said, you may cause panic. People genuinely don't know how to respond when they hear an alarm in an unfamiliar place.

And that's if you're lucky. I once had to deal with people who did not want to evacuate a building when a fire alarm went off. And when I got them out the door I had trouble convincing them to leave the building's deck. Some people seem to think that it is okay to ignore a fire alarm, even a probable false alarm, in a wood structure with a wood deck.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by db48x on Sunday November 15, 2026 @07:24PM (#50512464)
It wouldn’t need to be that loud. Unlike a fire that becomes dangerous in mere minutes, this type of accident takes days to unfold. Most automatic pool systems will have a light or a beeping noise to let the operator of a failure. The email is just a convenience. The hotel staff should have been checking the system directly every morning to see if there were errors before they opened the pool for the day.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by wongarsu on Monday November 16, 2026 @03:42AM (#50515418)
If you only need employees to react you can replace the rotating red light and klaxon with a smaller flashing white light that plays a pleasant but slightly annoying "ding" sound every five seconds

It just has to be annoying enough that even the most junior employee would radio in their supervisor to get it to stop

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by trollbridge on Sunday November 15, 2026 @05:56PM (#50511782)
We have a general design principle that we assume important emails will not be delivered, will not be read, and will not be actioned on.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by helterskelter2 on Monday November 16, 2026 @01:18AM (#50514610)
I generally ignore emails until I get bugged about something 2-3 times or the subject line is all caps.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by gregglain on Sunday November 15, 2026 @04:13PM (#50510814)
First this is so sad and preventable. I would be interested in the "follow the money" on this.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by perching_aix on Sunday November 15, 2026 @04:22PM (#50510909)
[delayed]
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by ponector on Sunday November 15, 2026 @05:23PM (#50511481)
In real life shared responsibility means no one is responsible for it.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by perching_aix on Sunday November 15, 2026 @05:34PM (#50511600)
Hah, fun that you bring that up. Just had quite the struggle articulating to management a few weeks ago that "shared ownership" runs contrary to the very idea of ownership, which I guess is basically the same topic just with different words. Either we take shares ownership seriously and actually organize and form consensus on things, or it's just bullshit. Naturally, did not really lead anywhere. So you definitely have a point there.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by ponector on Monday November 16, 2026 @03:20AM (#50515277)
Shared ownership is perfect: if project succeeded you can claim you've contributed heavily to the success. If project failed - you did everything toward success, but other team members failed.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by delta_p_delta_x on Sunday November 15, 2026 @06:21PM (#50511958)
> Decades of “root cause analysis” poisoning have ruined our ability to think about safety and reliability holistically.

This is not the case in aviation. Even in the most egregiously obvious single-cause failures, there's always an analysis of issues in the wider system that allowed that single cause to become the single cause in the first place.

I agree with your points, though. In this case, Microsoft absolutely takes shared responsibility. It is only a matter of time until a similar outage in some other wider networked system with a so-called 'five/seven/nine nines uptime guarantee' goes out for a couple of days, and takes with it some critical life-sustaining system, and people die.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by thephyber on Sunday November 15, 2026 @08:21PM (#50512886)
This seems wrong (obviously I don't have any more info than you).

Ford doesn't build all of the modules of their cars. They design specs and outsource manufacturing. But legally speaking, Ford is responsible for design, testing, and support / recalls. It's their responsibility to contract in the terms that both protect their customers' safety and Ford's liability due to spec / contract violations.

When your company is responsible for the health of staff and guests while dealing with deadly gasses, it's your company's responsibility to ensure that all of your vendors detail how their failure modes fail safe.

This is the same exact logic that has been migrating software companies to detail their risk by demanding their vendors all supply a "Software Bill of Materials" (which is very difficult in practice, but is necessary for the downstream users to protect themselves from upstream risk).

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by notatoad on Sunday November 15, 2026 @10:01PM (#50513510)
the point of root cause analysis isn't to find the one person or entity at fault. the point is to determine what you can fix, so this doesn't happen again next time.

anything that was "at fault" that you can't do anything about doesn't matter. and you can't do anything about a microsoft outage, all you can do is design around it.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by wallst07 on Monday November 16, 2026 @01:23AM (#50514636)
Run the 5-Whys on this, start with:

"Why did people get gas poisoning"

https://en.wikipedia.org/wiki/Five_whys

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by ponector on Monday November 16, 2026 @03:22AM (#50515283)
Because the are too much alive. AI robots cannot get gas poisoning. Case closed!
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by thaumasiotes on Monday November 16, 2026 @04:13AM (#50515583)
"Five Whys" is never going to mean anything as a methodology. It works if competent people use it and it fails if incompetent people use it. But pretty much everything works if you're free to assume that everyone involved is competent.

The point of a process is to constrain incompetent people into doing something that will work despite their incompetence. Five Whys is entirely subjective and can never constrain anyone; it's just something that some people think sounds deep.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by toofy on Sunday November 15, 2026 @03:39PM (#50510461)
> This is 100% fault of the hotel, not of Microsoft.

yeah, these companies need to be told to stop relying on cloud services for everything.

its wild to me how many criticism things that absolutely do not need the internet are relying on the internet.

much blame to go around here:

- whoever designed the stupid alert system.

- the marketing departments in countless companies ignoring engineers when they market to normie customers “connect everything”

- the hotel company for seemingly doing no dilligence at all. and just blindly listening to some salesman “connect everything”

- accountant types and their “we can save pennie’s if we cut all of these corners. and the stupid corners they choose to cut.

- etc…

on a macro scale it’s amazing to me how many people are victims to slimey car salesman types leading to situations like this.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by mitxela on Sunday November 15, 2026 @07:54PM (#50512701)
It's unlikely that the hotel themselves manages the pool. They pay a pool company to keep their pool running.

5/5

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by dwaltrip on Sunday November 15, 2026 @08:39PM (#50512998)
[delayed]
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by globular-toast on Monday November 16, 2026 @02:00AM (#50514847)
It has nothing to do with the cloud. For safety critical systems you don't rely on any single thing, cloud or otherwise.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by raxxorraxor on Monday November 16, 2026 @03:00AM (#50515185)
True. This is why the cloud should never be a requirement for anything safety critical. Or if you make that bad choice, design your system with a sensible offline operation mode.

Regardless of high availability promises of cloud services, a local internet outage will supersede the risk and is a risk factor in most places of the world (probably everywhere). Risk that nobody aside the local ISP could quantify either. So logically any system critically depending on the cloud and isn't designed with an offline fallback has unmitigated risk factors.

Sure, if you host some online service, this risk cannot be mitigated. But it can for everything else. And if you need an offline operation mode, in most cases the non-cloud design is simply superior. So cloud is synonymous with having a single point of failure?

This case is one where on risk analysis was done at all or this would have surfaced pretty quickly. There are problems too if a pool doesn't get chlorinated, but it is probably better than to poison your guests.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by mrweasel on Monday November 16, 2026 @04:10AM (#50515564)
Also in this case, it's people completely misunderstanding email. Way to many basically view email as a messaging platform, but with a slight lag. I'm apparently old, because I remember reading that emails could take seven days to arrive and that would be within spec.

Knowing that your email could be delayed by days changes how you view it and the applications you deem it suitable for.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by madaxe_again on Sunday November 15, 2026 @03:44PM (#50510524)
This gives me an idea for installation art:

A grand piano is suspended from the ceiling with an electromagnet.

The electromagnet works as long as there is power.

No special battery backups, anything.

Guests are invited to stand underneath it, after signing a waiver.

That’s it. That’s the art.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by juujian on Sunday November 15, 2026 @04:04PM (#50510741)
Please do it, I would spend ~4$ to do that.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by Dylan16807 on Sunday November 15, 2026 @07:44PM (#50512620)
Is the joke that the name is wrong? I'm pretty sure no power grids reach 5 nines of uptime.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by jrm4 on Sunday November 15, 2026 @04:02PM (#50510719)
The law is smarter than this, thankfully.

Your second sentence is 100% correct. Your first is questionable; Microsoft, a kabillion dollar company has, for years, held itself out to be reliable and safe. And it repeatedly has shown that it is not. That opens them up to some liability here; and frankly deserves to be better held to the standards it (either directly or indirectly) claims.

And see e.g. "Implied warranty of merchantability" as a refuation of the possibility that you can wave this away with a Terms of Service.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by teraflop on Sunday November 15, 2026 @06:33PM (#50512045)
Well, notwithstanding the fact that this incident happened in Germany...

> And see e.g. "Implied warranty of merchantability" as a refuation of the possibility that you can wave this away with a Terms of Service.

"Implied warranty of merchantability" is a term of art in United States law, and it applies only to tangible goods, not to services.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by Hizonner on Sunday November 15, 2026 @07:33PM (#50512536)
What exposure I have to German law leads me to suspect that their equivalent concepts are not forgiving on this kind of issue.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by crazygringo on Sunday November 15, 2026 @04:24PM (#50510941)
Seriously, what's next -- hotels which decide that smoke detectors ought to e-mail the front desk rather than blaring a loud alarm?

I'm surprised an e-mail only alert is even legal for this. Or is chlorine level monitoring not something that is regulated?

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by al_borland on Sunday November 15, 2026 @05:27PM (#50511512)
It's easy to say this in hindsight, but how many pools/hot tubs have employees monitoring an alert panel around the clock for water quality metrics of the water?

Countless systems are relying on email as a means to alert people to issues, for better or worse. Having a physical alarm for everything is rarely practical for a myriad of reasons.

I worked in a data center 20 years ago that had alarms on the wall for when the halon was preparing to be released for fire suppression. There are likely dozens of things that could be monitored, but it was only the catastrophic thing that once it started, there was no stopping it, that got lights on the wall.

Too many loud alarms make people start ignoring alarms. I would guess that there is a fairly large margin of error in pools before there is a health risk, so it likely wasn't deemed alarm worthy. As someone who's been in a pool before, I'd also imagine that people would notice and say something, or it would be unpleasant enough to not want to spend time in it. That's what I find most surprising... that people weren't immediately turned off just being around it, to the point of staying out.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by bmitch3020 on Sunday November 15, 2026 @06:15PM (#50511921)
On every public pool I've ever been to, there was at least one lifeguard on duty. And part of that guard's duties included manually checking the chemical levels. Automation is great, but there still needs to be a human present that can verify those systems are working at the start of their shift, and to be somewhere they can hear or see any alarms if they go off.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by swolios on Monday November 16, 2026 @12:18AM (#50514323)
almost all hotel pools have a sign " no lifeguard on duty " as no-non mega corp hotel (and even those dont) have lifeguards.

Maybe a waterpark resort, but hotels no. These days they barely afford a front desk employee and a cleaner.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by trollbridge on Sunday November 15, 2026 @05:55PM (#50511773)
100% correct. An email service is not to be used for critical alerting and communication. There are other systems that are up to the task - including infrastructure built on Azure which could have the required level of fault tolerance in it.
Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by damowangcy on Sunday November 15, 2026 @11:35PM (#50514083)
Outside tech, the further you are away from Silicon Valley (hint: not physically but industry wise), the more common it is to see such design of systems. Why? Because technology is a cost and IT department is a cost center which doesn't increase revenue.

When an incident like this happened, they might add new "system" to improve the situation, probably in the lines of having someone to patrol and check off a list routinely.

Re:Microsoft 365 outage leads to chlorine gas poisoning of 1... by sieste on Monday November 16, 2026 @03:40AM (#50515406)
"Sorry all those kids burned to death on the rollercoaster, the fire alarm must have gone to spam."