> At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job.
That doesn’t match my experience at all.
In the places I've experienced with poor management and engineering cultures, what I've seen is management dictates all sorts of horrible, asinine things (like story points being treated as deadlines, etc.) that leads directly to problems like this because engineering is not given the space it need to create robust solutions in the first place.
The prevailing winds are, "Just get it done." The outcome is constant fire fighting and triage because of simple problems like bad data finding it's way into the system.
That happens. But not every time, and even "usually" requires some statistics collecting to be sustained.
Very often people just say "yes, we learned an expensive lesson", and change something that may or may not help the next time it happens.
If the answer to the first question is a bunch of finger pointing or the answer to the second is along the lines of "nothing", heads will likely roll.
My first boss as a FTE at a trading shop made a change to a production database outside of process before a weekend. Come Monday morning, production trading systems couldn't come up for market open. We were also an options market maker. He refused to accept responsibility for his actions and denied there was anything done to prevent it. He was quickly behind closed doors in his boss's office. I had to undo the changes he made the week before (hooray for audit tables) as soon as I could. We were potentially liable for a substantial penalty from the exchange for being out of the market, and every minute mattered. Don't know if we ever did get penalized, but he was fired and out the door by 10AM that day.
I've certainly seen and made lots of mistakes over the decades since, but that's the only one for which I've seen someone canned. Response matters.
That part doesn’t match my experience at all.
Good question though..
Oh, and the rampant racism and classism, thanks to the standard practice of including a photo on one's resume...
Probably not in this case, but you couldn't assume for every...
Maybe the CEO in the book gets his due?
Blaming a contractor? To save face? Isn't saving face. It's an embarrassment.
TBH IDK the book was mentioned on HN and I figured I'd check it out.
Exactly! There wasn't any real fallout for Southwest, who had a shutdown less than a month ago, for similar outdated technology reasons.
There's running lean and then there is running recklessly.
I joke but the right amount of bureaucracy is good, but people are very careful about adding more of it because you can’t undo it
For example, running "/some/script > data-file" instead of "/some/script < data file" still fits the very generic wording the FAA is sharing publicly.
Could have been accidentally overwriting some idx/dat pair, or some raw database file, etc. Though, yeah, there should still be some controls, troubleshooting tools, logging, etc, that would have made things more clear and recoverable.
It explains that procedures were circumvented and something in production was edited - something that was not allowed.
You can put as many procedures in place as you want, but unless the humans follow these procedures (I.e. not circumvent them deliberately) then someone always has some level of access where they can cause mayhem.
Sure you can programmatically enforce this to a large extent, but ultimately there are always some humans with prod access, or BGP editing access, or firewall write access, or access to that validation logic itself! And it requires that the humans follow the organisational procedures in place.
From what the article says, it sounds like the humans deliberately circumvented procedures, in which case, it is the engineers fault, and they should be disciplined.
> Like other computer systems that are critical to operating flights, the FAA has imposed procedures to ensure data aren’t damaged by technicians working on them, said the person. The file or files were altered in spite of rules that prohibit those kind of changes on a live system.
> Agency officials are attempting to determine whether the two people made the changes accidentally or intentionally, and if there was any malicious intent, the person said.
It is clearly implied that had the rules been followed, the corruption wouldn't have occurred.
Your blame-the-FAA scenario where the rules were incomplete or flawed isn't ruled out, but I don't see value in asserting it without evidence. I think it's perfectly plausible that someone just cheated a bit. That's hardly a rare situation in any IT environment, and so it seems like Occam favors the article's interpretation.
It can be justified as "the instructions are out of date", "this is an urgent change", "just get it done!", etc.
Heck, sometimes following the written rules is derided as "working to rule" or "takes longer than others to complete the task".
I'm just saying that blame for these issues only sometimes belongs on the line workers.
In large org thinking, if something bad happens, and you add a rule, then it's fixed.
The problem is that this is not compatible with human psychology.
This leads to many rules, the rules are unstructured and impossible to learn, the rules probably also contradict each other.
In my experience, a rule without automatic enforcement should be the absolute last thing to depend on. If you do this, your org is broken.
The Toyota Production System with it's blame on process design instead of humans is still the best way to avoid these issues IMO.
Do 5-whys where human error is not allowed as a root cause. Design your process with poka yoke in mind. If somebody can forget something they will. Don't depend on it.
If that sort of thing actually happened a lot more C level people would have been fired for over-hiring, but that rarely happens.
I don't think this is a thing that actually happens.
But the previous CEO looks like he was the spreadsheet type and rejected these kinds of ideas. So who is to blame?
(Various elements and automations of the system have been created and updated, so these could have included improvements in design.)
> […] a data file was damaged as a result of a failure to follow government procedures, […]
The more critical processes humans are expected to perform, the worse the expected results.
That goes 10x for outside or temporary “contractors” who are not going to be experts in a systems unique, esoteric, or unexpectedly irresponsible points of fragility
Second, nothing would happen at a private company because they would involve marketing and marketing wouldn’t blame anyone in a press release. The only difference with FAA is that they didn’t let a marketing department write their release.
But really the real issue here is that everyone is posting to make wide generalizations from a one time event.
Can you share an example of where that has actually happened?
I doubt that. More likely: the defect would be identified and fixed, someone would start working on a training-wheels editor for the file to ensure the problem doesn't happen again. Any change in revenues or profits would be used by the CEO to drive change in whatever thing needs changing, and the whole incident would be forgotten. Source: have owned four private companies. If everyone is fired immediately, no one learns a lesson and everyone is terrified to give leaders bad news.
The engineering precautions in system design always involve trade-offs. There is no perfect system that could be created for acceptable costs. Ultimately it is always going to come down to humans following the rules, hopefully as few of those very-trustworthy humans as possible, though that's where the FAA may have been lax.
Sec. Buttigieg apparently agrees: https://www.nbcnews.com/news/us-news/software-blamed-faa-out...
> Transportation Secretary Pete Buttigieg told NBC News that he has asked the FAA, "to make sure that there are enough safeguards built into the system that this level of disruption can't happen because of an individual person’s decision or action or mistake."
People in senior management positions are at fault for creating a broken org and product. At a private company a failure like this would see the CTO immediately fired and even the CEO having to beg the board for forgiveness and keeping their job. The fault is not with the two contractors, it's with the bad engineering that enables a mistake to destroy the product.