News for hackers. Stuff that matters.
Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by Aurornis on Wednesday November 19, 2025 @05:04PM (#45985937)
Thanks for the summary for those of us who can't watch video right now.

There are so many layers of failures that it makes you wonder how many other operations on those ships are only working because those fallbacks, automatic switchovers, emergency supplies, and backup systems save the day. We only see the results when all of them fail and the failure happens to result in some external problem that means we all notice.

Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by arjie on Wednesday November 19, 2025 @05:19PM (#45986108)
It seems to just be standard "normalization of deviance" to use the language of safety engineering. You have 5 layers of fallbacks, so over time skipping any of the middle layers doesn't really have anything fail. So in time you end up with a true safety factor equal only to the last layer. Then that fails and looking back "everything had to go wrong".

As Sidney Dekker (of Understanding Human Error fame) says: Murphy's Law is wrong - everything that can go wrong will go right. The problem arises from the operators all assuming that it will keep going right.

I remember reading somewhere that part of Qantas's safety record came from the fact that at one time they had the highest number of minor issues. In some sense, you want your error detection curve to be smooth: as you get closer to catastrophe, your warnings should get more severe. On this ship, it appeared everything was A-OK till it bonked a bridge.

Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by bombcar on Wednesday November 19, 2025 @06:09PM (#45986578)
This is the most pertinent thing to learn from these NTSB crash investigations - it's not what went wrong at the final disaster, but all the things that went wrong that didn't detect that they were down to one layer of defense.

Your car engaging auto brake to prevent a collision shouldn't be a "whew, glad that didn't happen" and more a "oh shit, I need to work on paying attention more."

Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by dmurray on Wednesday November 19, 2025 @07:45PM (#45987343)
Why then does the NTSB point blame so much at the single wiring issue? Shouldn't they have the context to point to the 5 things that went wrong in the Swiss cheese and not pat themselves on the back with having found the almost-irrelevant detail of

> Our investigators routinely accomplish the impossible, and this investigation is no different...Finding this single wire was like hunting for a loose rivet on the Eiffel Tower.

In the software world, if I had an application that failed when a single DNS query failed, I wouldn't be pointing the blame at DNS and conducting a deep dive into why this particular query timed out. I'd be asking why a single failure was capable of taking down the app for hundreds or thousands of other users.

Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by toast0 on Wednesday November 19, 2025 @08:18PM (#45987601)
The faulty wire is the root cause. If it didn't trigger the sequence of events, all of the other things wouldn't have happened. And it's kind of a tricky thing to find, so that's an exciting find.

The flushing pump not restarting when power resumed did also cause a blackout in port the day before the incident. But you know, looking into why you always have two blackouts when you have one is something anybody could do; open the main system breaker, let the crew restore it and that flushing pump will likely fail in the same way every time... but figuring out why and how the breaker opened is neat, when it's not something obvious.

Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by nothercastle on Wednesday November 19, 2025 @09:56PM (#45988353)
Operators always like to just clear the fault and move on they have extremely high pressure to make schedule and low incentive to work safely
Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by plorg on Wednesday November 19, 2025 @08:33PM (#45987707)
That seems like a difference between the report and the press release. I'm sure it doesn't help that the current administration likes quick, pat answers.

The YouTube animation they published notes that this also wasn't just one wire - they found many wires on the ship that were terminated and labeled in the same (incorrect) way, which points to an error at the ship builder and potentially a lack of adequate documentation or training materials from the equipment manufacturer, which is why WAGO received mention and notice.

Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by bombcar on Wednesday November 19, 2025 @09:03PM (#45987949)
It’s also immediately actionable and other similar ships can investigate their wires
Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by da_chicken on Thursday November 20, 2025 @07:27AM (#45991889)
> I'm sure it doesn't help that the current administration likes quick, pat answers.

Oh, the wire was blue?

In all seriousness, listing just the triggering event in the headline isn't that far out of line. Like the Titanic hit an iceburg, but it was also traveling faster than it should in spite of iceberg warnings, and it did so overloaded and without adequate lifeboats, and it turns out there were design flaws in the hull. But the iceberg still gets first billing.

Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by plorg on Thursday November 20, 2025 @09:47AM (#45993092)
I think it reads as too cute by a half. The wire was just the one of dozens of problems that happened last. It's natural to attribute cause in that way, but it's not really helpful in communicating the purpose of these investigations.

If this represents a change in style and/or substance of these kinds of press releases, my hunch would be that the position was previously hired for technical writers but was most recently filled by PR.

Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by f4c39012 on Thursday November 20, 2025 @12:17PM (#45995030)
Interesting, recent podcast on the subject https://99percentinvisible.org/episode/632-the-titanics-best...
Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by aidenn0 on Thursday November 20, 2025 @12:47AM (#45989371)
I had to disable the auto-brake from RCT[1] sensors because of too many false-positives (like 3 a week) in my car.

1: rear-cross-traffic i.e. when backing up and cars are coming from the side.

Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by ecshafer on Thursday November 20, 2025 @10:33AM (#45993676)
One of my car's auto-brake sensors triggers when I back up out of by drive way. I can not back out of my drive way with the sensor on.
Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by raverbashing on Thursday November 20, 2025 @03:10AM (#45990191)
Yes and having 3 O-rings doesn't mean you can have one frozen solid "just this time"
Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by esafak on Thursday November 20, 2025 @08:55AM (#45992575)
Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by esafak on Thursday November 20, 2025 @08:54AM (#45992557)
The solution then is observability, to use the computing term; to know the state of every part of the system.
Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by crote on Wednesday November 19, 2025 @05:17PM (#45986084)
Oh, it gets even worse!

The NTSB also had some comments on the ship's equivalent of a black box. Turns out it was impossible to download the data while it was still inside the ship, the manufacturer's software was awful and the various agencies had a group chat to share 3rd party software(!), the software exported thousands of separate files, audio tracks were mixed to the point of being nearly unusable, and the black box stopped recording some metrics after power loss "because it wasn't required to" - despite the data still being available.

At least they didn't have anything negative to say about the crew: they reacted timely and adequately - they just didn't stand a chance.

Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by nothercastle on Wednesday November 19, 2025 @09:58PM (#45988370)
It’s pretty common for black boxes to be load shed during an emergency. Kind of funny how that was allowed for a long time.
Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by MengerSponge on Wednesday November 19, 2025 @11:12PM (#45988871)
"they reacted timely and adequately" and yet: they're indefinitely restricted (detained isn't the right word, but you get it) to Baltimore, while the ship is free to resume service.
Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by dboreham on Wednesday November 19, 2025 @09:22PM (#45988106)
Also the zeroth failure mode: someone built a bridge that will collapse if any of the many many large ships that sail beneath it can't steer itself with high precision.
Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by foobar1962 on Wednesday November 19, 2025 @10:14PM (#45988483)
Ships were a lot smaller when the bridge was designed and built.
Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by fl7305 on Thursday November 20, 2025 @12:16PM (#45995024)
In 1971 there where ships with almost twice the displacement of the Dali.

They weren't freight ships destined for Baltimore, but it wasn't hard to imagine future freight ship sizes when designing the bridge in the early 1970s.

The London sewer system was designed in the 1850s, when the population was around two million people.

It was so overdesigned that it held up to the 1950s, when the population was over 8 million. It didn't start to become a big problem until the 1990s.

Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by myself248 on Thursday November 20, 2025 @06:56AM (#45991714)
Right? There's an artificial island in that very harbor, which could be rammed by similar ships all day and give nary a fuck. It's called Fort Carroll and it was built in the *1850s*.

Why the bridge piers weren't set into artificial islands, I can't fathom. Sure. Let's build a bridge across a busy port but not make it ship-proof. The bridge was built in the 1970s, had they forgotten how to make artificial islands?

Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by potato3732842 on Thursday November 20, 2025 @08:34AM (#45992388)
If you design a fort and it actually gets used and turns out to suck that WILL be the end of your career in the military even if it only comes out as sucking 20yr later unless you have an airtight case why it's not your fault. That's just how the .mil works. Heads MUST roll. This is completely the opposite from big company bureaucracy and on a literal different planet than civil government bureaucracy.

The organizations that made the bridge happen were so much more vast and so much higher turnover and subject to way, way, way looser application of consequences than the one that built the fort it would be literally impossible to get them to produce something so unnecessarily robust for the average use case.

This sort of "everything I depend on will just have to not suck because my shit will keel right over if it sucks in the slightest" type engineering is all over the modern world and does work well in a lot of places when you consider lifetime cost. But when it fails bridges fall over and cloudflare (disclaimer, didn't actually read that PM, have no idea what happened) goes down or whatever.

Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by prewett on Thursday November 20, 2025 @11:40AM (#45994572)
> on a literal different planet than civil government

Unless the military was relocated to Mars (or at least the Moon) during the shutdown, I think the word is "metaphorically" instead of "literal".

Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by B1FF_PSUVM on Thursday November 20, 2025 @12:07PM (#45994888)
Or it was just a different plane ...
Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by haddonist on Wednesday November 19, 2025 @11:43PM (#45989045)
One of the things Sal Mercogliano stressed is that the crew (and possibly other crews of the same line) modified systems in order to save time.

Rather than doing the process of purging high-sulphur fuel that can't be used in USA waters, they had it set so that some of the generators were fed from USA-approved fuel, resulting in redundancy & automatic failover being compromised.

It seems probable that the wire failure would not have caused catastrophic overall loss of power if the generators had been in the normal configuration.

Re:Loose Wire Leads to Blackout, Contact with Francis Scott ... by myself248 on Thursday November 20, 2025 @06:52AM (#45991687)