News for hackers. Stuff that matters.
Re:FAA says computer failure was caused by 2 contractors who... by sokoloff on Saturday January 14, 2023 @11:30AM (#34381500)
I agree that the ultimate root fault isn’t the recent data entry.

> At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job.

That doesn’t match my experience at all.

Re:FAA says computer failure was caused by 2 contractors who... by scruple on Saturday January 14, 2023 @11:40AM (#34381580)
Same here.

In the places I've experienced with poor management and engineering cultures, what I've seen is management dictates all sorts of horrible, asinine things (like story points being treated as deadlines, etc.) that leads directly to problems like this because engineering is not given the space it need to create robust solutions in the first place.

The prevailing winds are, "Just get it done." The outcome is constant fire fighting and triage because of simple problems like bad data finding it's way into the system.

Re:FAA says computer failure was caused by 2 contractors who... by toomuchtodo on Saturday January 14, 2023 @11:39AM (#34381577)
Right. Engineer closest to the fault gets scapegoated and fired, new hire gets hired in without any idea what they’re coming into, and the show goes on.
Re:FAA says computer failure was caused by 2 contractors who... by marcosdumay on Saturday January 14, 2023 @01:26PM (#34382580)
> Engineer closest to the fault gets scapegoated and fired

That happens. But not every time, and even "usually" requires some statistics collecting to be sustained.

Very often people just say "yes, we learned an expensive lesson", and change something that may or may not help the next time it happens.

Re:FAA says computer failure was caused by 2 contractors who... by toomuchtodo on Saturday January 14, 2023 @02:00PM (#34382937)
Really just depends on the org, and you’re not going to know until you’re on the inside.
Re:FAA says computer failure was caused by 2 contractors who... by marcosdumay on Saturday January 14, 2023 @02:02PM (#34382963)
Yep, I completely agree with that. It can change from team to team too. Also, it can suddenly change at any time without a warning.
Re:FAA says computer failure was caused by 2 contractors who... by hermitdev on Saturday January 14, 2023 @02:35PM (#34383339)
I think it also matters on what the response to the incident was, in particular the answer to the questions of: who was responsible and how can we prevent this from happening again?

If the answer to the first question is a bunch of finger pointing or the answer to the second is along the lines of "nothing", heads will likely roll.

My first boss as a FTE at a trading shop made a change to a production database outside of process before a weekend. Come Monday morning, production trading systems couldn't come up for market open. We were also an options market maker. He refused to accept responsibility for his actions and denied there was anything done to prevent it. He was quickly behind closed doors in his boss's office. I had to undo the changes he made the week before (hooray for audit tables) as soon as I could. We were potentially liable for a substantial penalty from the exchange for being out of the market, and every minute mattered. Don't know if we ever did get penalized, but he was fired and out the door by 10AM that day.

I've certainly seen and made lots of mistakes over the decades since, but that's the only one for which I've seen someone canned. Response matters.

Re:FAA says computer failure was caused by 2 contractors who... by drstewart on Saturday January 14, 2023 @02:00PM (#34382929)
>Right. Engineer closest to the fault gets scapegoated and fired, new hire gets hired in without any idea what they’re coming into, and the show goes on.

That part doesn’t match my experience at all.

Re:FAA says computer failure was caused by 2 contractors who... by pleb_nz on Saturday January 14, 2023 @02:53PM (#34383530)
You can just about tell what parts of the world a poster comes from when they start throwing the word 'fire' around willy nilly.

Good question though..

Re:FAA says computer failure was caused by 2 contractors who... by KennyBlanken on Saturday January 14, 2023 @03:55PM (#34384085)
You can tell what part of the world a poster comes from when they're smugly pretending that companies still don't have plenty of levers to pull to push you out the door, doing so by making you miserable.

Oh, and the rampant racism and classism, thanks to the standard practice of including a photo on one's resume...

Re:FAA says computer failure was caused by 2 contractors who... by bennyelv on Saturday January 14, 2023 @12:57PM (#34382263)
It might not be the CTO's fault at all - maybe they inherited it, identified the issues, and are getting them solved.

Probably not in this case, but you couldn't assume for every...

Re:FAA says computer failure was caused by 2 contractors who... by chiefalchemist on Saturday January 14, 2023 @01:23PM (#34382551)
Funny. I'm reading The Phoenix Project and just finished the about the CEO chewing out various employees. I keep thinking, responsiblity flows upstream, or should. Ultimately it is the CEO's job to put his people and the project in a position to suceed.

Maybe the CEO in the book gets his due?

Blaming a contractor? To save face? Isn't saving face. It's an embarrassment.

Re:FAA says computer failure was caused by 2 contractors who... by noughtme on Saturday January 14, 2023 @06:38PM (#34385225)
Still looks interesting, but I was hoping The Phoenix Project was a post-mortem of the Phoenix Pay System (scandal) https://en.m.wikipedia.org/wiki/Phoenix_pay_system
Re:FAA says computer failure was caused by 2 contractors who... by chiefalchemist on Saturday January 14, 2023 @06:43PM (#34385263)
It may be "inspired by" as in the book TPP is among other things about the stores POS.

TBH IDK the book was mentioned on HN and I figured I'd check it out.

Re:FAA says computer failure was caused by 2 contractors who... by fineIllregister on Saturday January 14, 2023 @01:40PM (#34382728)
> That part doesn’t match my experience at all.

Exactly! There wasn't any real fallout for Southwest, who had a shutdown less than a month ago, for similar outdated technology reasons.

Re:FAA says computer failure was caused by 2 contractors who... by Someone1234 on Saturday January 14, 2023 @02:50PM (#34383493)
Maybe even more pertinent is that they had a similar one in 2016 (smaller scale, but similar cause/effect) and then didn't fix anything, so it could happen again recently.

There's running lean and then there is running recklessly.

Re:FAA says computer failure was caused by 2 contractors who... by CamperBob2 on Saturday January 14, 2023 @08:05PM (#34385724)
"Move slowly and break things!"