> There were many things wrong with this project, but one of them was fatally bad code, which this guy took money to write.

If I remember correctly, the bug in this case was a race condition between "normally" running code and an interrupt handler. The race was only triggered if an interrupt happened in just the right window between two instructions. I'd be willing to bet that 99% of programmers, if simply given the code and asked, "Is there a bug here?" would have answered "No".

Should people writing operating-system-level medical equipment software be required to have a basic training about race conditions and how to prevent them? Yes. Is it fair to expect a random engineer who was ordered by his company to rewrite the code to work without a hardware interlock to know what he doesn't know? No.

> Is it fair to expect a random engineer who was ordered by his company to rewrite the code to work without a hardware interlock to know what he doesn't know? No.

Looking back at when I did Safety Critical Systems at Uni way back in another era, the most important points that we got out of the Therac-25 case study were not to do with bugs at all, but to do with the deficiencies of the system architecture and methodology, especially the decisions that led up to it.

It is 100% fair to expect people working on life-critical systems to either a) know that they are competent to work on life-critical code, or b) to not work on life-critical code.

The guy wasn't ordered to do this at the point of a gun. His bosses asked him to do something, he said yes because he liked the money, and people died. The bosses are also responsible, but that doesn't mean that his negligence didn't kill people.

Is said "random engineer" an engineer? No. He's a code monkey and should be expected to be paid like a code monkey.

Obviously we don't know who wrote the code, so we can't ask him, but did he even know the consequences of what he was writing? Was he even aware of the potential lethality?

The very fact that we don't know who wrote the code is exactly why it doesn't matter. The complete lack of traceability and accountability is what caused these deaths.

There's so many things that went wrong in order for Therac 25 to kill people that it's irrelevant who wrote the code. It could've just as easily been you or me.

> Was he even aware of the potential lethality?

It could be worse than that. He might not have even known the actual purpose and complete function of the system for which he was writing code.

That's common when big projects get outsourced. One might call it "decomposing the problem into small tasks" but it ends up with people writing code for machinery that they have no knowledge of. I've seen this myself. These kinds of projects depend upon layers and layers of oversight and process.

I think that the folks who are looking for "a perpetrator" are utterly missing the key-value of studying the Therac-25 case study.

I am personally not looking for "a perpetrator". Many people share responsibility for a failure like this. But that's all the more reason that we should clearly describe who did what, when, and why.

Please note that you are working very hard to make up a fantasy situation that entirely exonerates the person like you. Your fantasy does not match the known facts. Think hard about why you think that's the most important thing for you to do here.

He knew or should have known the potential consequences of the code he was writing. Maybe this could have been you, but it definitely wouldn't have been me, because I would not have committed the first line of code until I was reasonably sure that a) I was not going to kill somebody accidentally, and b) the project had sufficient safeguards.

I'd also like you to note that you're pointing out the lack of accountability as a problem, and then dismissing one of the fundamental mechanisms of accountability: naming people who harm others.

Yes, other people on the project also should have pushed for traceability and accountability. But responsibility is not zero sum. "I was just following orders" is never an excuse for killing people.

A culture of blame leads to programmers of safety critical software hiding their mistakes.

On a project? Sure. I'm a big fan of blameless retrospectives as a way to maximize quality. But that only works in the context of a quality-oriented culture that is strongly focused on continually improving safety.

However, a) none of that was true here, and b) in the wider societal frame, I think blamenessless must not always trump accountability. When we're talking about egregious negligence leading to death, I think naming the culprits is the very least we should do.

I can see why you would say that, but isn't classical engineering a counter example? Professional engineers have a culture of accountability and responsibility. No system is perfect, but (to my eyes at least) classical engineering works safely without a culture of blame.

Preferable to the "mistakes were made" fog bank of diffuse nonblame. People respond to incentives whether cash or heads on pikes.

A culture of refusing responsibility is better how?

The opposite of a blame culture is not a culture of refusing responsibility, but a system where you look for problems in the process when a fault makes it into the final product. It's always a mixture of specification, implementation, and QA that makes mistakes possible. Putting the blame on the engineer is not very helpful, it does little to prevent future problems. Trying to figure out how it was possible that an implementation bug made it through QA allows you to prevent similar mistakes in the future (at least if you're lucky).

Just to be clear, I'm not putting the blame on the engineer. Many people are responsible for this failure. What I'm saying is that the engineer is one of the responsible people, and that as fellow engineers we should be especially concerned with making sure that responsibility is correctly performed and has appropriate accountability.

But why you want to keep him accountable and not the ones who did the testing and released the product?

I never said he should be the only one. Responsibility is not zero sum. But as a professional software developer, I am pointing out the professional responsibility we all have.

IMHO whoever decided to let out the hardware protection, present in older machines, is the 'culprit'.

Typical code cowboy mindset: expects all of the glory, none of the responsibility. Kindly piss off and don’t come back until you’ve learned proper systems design.

https://en.wikipedia.org/wiki/Fail-safe

Self-driving software might find its way around a controlled test track okay but is nowhere near fit to be in control on public roads. Safe way to test it the is to have human drive and have the software simulate the same drive, then compare the two for discrepancies. Rinse and repeat until the software consistently equals or betters human decision-making. Then you can consider putting the software in charge.

Programmers are supposed to be the technical experts in the room. So if they aren’t pushing back against ignorant uninformed hubristic Management’s bad decision-making and making it right, how the hell do they think their software will do any better? Fucking Worse than Useless, the whole cowardly bloody bunch.

I don't think you understand the basic concept let alone how the self driving systems are comprised and how large software development projects work. The uber car wasn't self driving. It required a human occupant to watch the road at all times. That is certainly not on the developers.

You appear to be laboring under the gross misassumption that the human is there to take control when the machine fucks up. They are not.

They are there to take the blame.

You cannot realistically expect a human spectator to switch modes at zero notice and save the machine from a deadly fuckup that the machine has already put them into. That’s not how human minds work. We just don’t bootstrap that fast. Jebus, we’re bad enough at salvaging bad situations we’ve knowingly manoeuvered ourselves into while already fully engaged in command and control mode.

At least when you—the human pilot—fucks up, you already know the decision chain that got you there because it is your own. With the machine you have first to determine it has gone catastrophically wrong, then determine how it has gone wrong, and finally calculate and execute the recovery strategy before…oh, whoops, too late: you just wiped out all the executive bonuses for this quarter. Also, there’s a blood streak on the street.

As for how large software development projects work, I think the trail of corpses already testifies more than suf as to how they don’t. Industrial institutionalization of incompetence is no defense, and any “professional” who hides behind it can go get fucked.

Exactly. Other companies knew that kind of attention and reaction was beyond human capacity, which is why they had two safety drivers. Which I'm still not sure is sufficient to compensate for the other sorts of negligence Uber staff indulged in here.

As much as I love the web, I think the move-fast-break-things ethos, which is arguably useful for startups doing who-cares-if-it-breaks things like social web front end tweaks, has been absolutely terrible for the industry more broadly. I have friends who make excellent money just sweeping up after the elephant parade of hotshots, solving infrastructure and code issues written by people who want to get paid like professionals without acting like ones. I'm glad for them, but the waste is maddening. And that's before we get to the body counts of places like Facebook and Uber.

That is not what the NTSB said.

Responsibility is not zero sum. The safety driver is responsible. But so are the people who sent out robot cars with a single safety driver, as they knew or should have known that paying attention in low-interaction situations for many hours in a row is not something humans reliably do.

The Uber managers and execs are also responsible, in that they set up the system that led to needless death.

But none of that absolves the programmers, some of whom did things that they knew or should have known were dangerous, and who did not make sure that the system they were committing code for was set up for proper safety.

Your notion that the only job of programmers is to meet the spec is one I deeply disagree with. We're not Amazon warehouse workers, desperate for a job and blindly following whatever orders come our way. We're highly paid professionals whose job is to understand what we're building and what effects it has. I think that's true for any sort of coding, but I believe it's very obviously true for life-critical systems. If we can't handle the responsibility, we shouldn't cash the (quite large) checks.

I agree, partly. They were testing the vehicle, and having a human in the driver's seat to take over in the event of a problem is a major component of that testing. That person likely became complacent in their duties due though, likely because the software seemed to work well enough.

The failure here was to expect someone to stay alert for a rare event for long periods of time. They probably should have had a more active task that would have kept them more alert.