Another interesting fact mentioned in the podcast is that the earlier (manually operated) version of the machine did have the same fault. But it also had a failsafe fuse that blew so the fault never materialized. Excellent demonstration of the Swiss Cheese Model: https://en.wikipedia.org/wiki/Swiss_cheese_model
> the earlier (manually operated) version of the machine did have the same fault. But it also had a failsafe fuse that blew so the fault never materialized.
#1 virtue of electromechanical failsafes is that their conception, design, implementation, and failure modes tend to be orthogonal to those of the software. One of the biggest shortcomings of Swiss Cheese safety thinking is that you too-often end up using "neighbor slices from the same wheel of cheese".
#2 virtue of electromechanical failsafes is that running into them (the fuse blew, or whatever) is usually more difficult for humans to ignore. Or at least it's easier to create processes and do training that actually gets the errors reported up the chain. (Compared to software - where the worker bees all know you gotta "ignore, click 'OK', retry, reboot" all the time, if you actually want to get anything done):
But, sadly, electromechanical failsafes are far more expensive then "we'll just add some code to check that" optimism. And PHB's all know that picking up nickles in front of the steamroller is how you get to the C-suite.
About a decade ago a rep from Videojet straight up lied to us about their 30W CO2 marking laser having a hardware interlock. We found out when - in true Therac-25 fashion - the laser kept triggering despite the external e-stop being active due to a bug in their HMI touch panel. No one noticed until it eventually burned through the lens cap. In reality the interlock was a separate kit, and they left it out to reduce the cost for their bid to the customer. That whole incident really soured my opinion of them and reminded me of just how bad software "safety" can get.
Blaming it on PHB's is a mistake. There were no engineering classes in my degree program about failsafe design. I've known too many engineers who were insulted by my insinuations that their design had unacceptable failure modes. They thought they could write software that couldn't possibly fail. They'd also tell me that they could safely recover and continue executing a crashed program.
This is why I never, ever trust software switches to disable a microphone, software switches that disable disk writes, etc. The world is full of software bugs that enable overriding of their soft protections.
BTW, this is why airliners, despite their advanced computerized cockpit, still have an old fashioned turn-and-bank indicator that is independent of all that software.
But one key component is that IF a failsafe is triggered, it needs to be investigated as if it killed someone; because it should NEVER have triggered.
Without that part of the cycle, eventually the failsafe is removed or bypassed or otherwise ineffective, and the next incident will get you.
The result is incredible safety.
Airplane reliability is from lots of failure analysis and work but also comprehensive maintenance plans and procedures.
The mechanical interlock essentially functioned as a limit outside of the control system. So you should build an ai system the same way- enforcing restrictions on the security agency from outside the control of the ai itself. Of course that doesn’t happen and devs naively trust that the ai can make its own security decisions.
Another lesson from that era we are re learning- in-band signaling. Our 2025 version of the “blue box” is in full swing. Prompt injection is just a side effect of the fact that there is no out of band instruction mechanism for llms.
Good news is - it’s not hard to learn the new technology when it’s just a matter of rediscovering the same security issues with a new name!
Wrong, any software failure can have huge consequences in someone's life, or company, by preventing some critical flow to take place, corrupting data related to someone's life, professional or medical record, preventing a payment on some specific goods that had to be acquired on that moment or never,....
It is business who requests features ASAP to cut costs and and then there are customers who don’t want to pay for „ideal software” but rather have every software for free.
Most devs and QA workers I know want to deliver best quality software and usually are gold plating stuff anyway.
Also, speaking out when the train is visibly going against a wall.
https://edition.cnn.com/2025/08/27/us/alaska-f-35-crash-acci...
If you want professional quality, we're the first line of actually making it happen, blaming others won't change anything.
https://ncees.org/ncees-discontinuing-pe-software-engineerin...
Does a construction engineer blame an architect's wacky designs if a building collapses? No, they either engineer it so it doesn't collapse, convince the architect that it will collapse because physics, or they refuse.
People want to be able to use a bridge for free too, doesn't mean there's no money in it.
As for gold plating, is that really improving software quality, or is that yak shaving / bike shedding?
That may seem a bit hypothetical but it can easily happen if you have a company that systematically underpays, which I'm sure many of us don't need to think hard to imagine, in which case they will systematically hire poor developers (because those are the only ones that ever applied).
It used to be that the poor performers (dangerous hip-shootin' code commitin' cowpokes) were limited in the amount of code that they could produce per time unit, leaving enough time for others to correct course. Now the cowpokes are producing ridiculous amount of code that you just can't keep up with.
My job hasn't significantly changed with AI, as AI generated code still has to pass all the hurdles I've set up while setting up this project.
But average construction worker is also average and average doctor as well.
World cannot be running on „best of the best” - just wrap your head around the fact whole economy and human activities are run by average people doing average stuff.
A lot of times that is boring meetings to discuss the simplification.
I can extend the same analogy to all the gen ai bs that’s floating around right now as well.
That’s not to say talent is unimportant, however, I’d need to see some real examples of high talent, no process, teams compared to low talent, high process, teams, then some mixture of the groups to make a fair statement. Even then, how do you measure talent? I think I’m talented but I wouldn’t be surprised to learn others think I’m an imbecile who only knows Python!
What I've been saying is methodology is mostly irrelevant, not that waterfall is specifically better than agile. Talent wins over the process but I can see how this idea is controversial.
I’d need to see some real examples of high talent, no process, teams compared to low talent, high process, teams, then some mixture of the groups to make a fair statement. Even then, how do you measure talent?
Yep, even if I made it my life's mission to run a formal study on programmer productivity (which I clearly won't) that wouldn't save the argument from nitpicking.
> Yep, even if I made it my life's mission to run a formal study on programmer productivity (which I clearly won't) that wouldn't save the argument from nitpicking.
I didn't ask for this, I just asked for sensible examples, either from your experience or from publicly available information.
That’s a horrible take. There is no amount of reviews, guidelines and documentation that can compensate for low quality devs. You can’t throw garbage into the pipeline and then somehow process it to gold.
I'd say this includes all of us all the time; a good developer never trusts their own work blindly, and spends more time gathering requirements and verifying their and others' work than writing code.
Despite all the procedures and tests, the software still managed to endanger the lives of the passengers.
https://en.wikipedia.org/wiki/Maneuvering_Characteristics_Au...
KC lost over $400 million in less than an hour due to an old feature toggle and a problem with their deployment process.
I suspect that few organizations that do all that, have a process/culture of ignoring bugs in the wild -- and those that do have such complicated domains that explaining the error is hard.
Software best practices today would probably also involve sending metrics, logs, error reports, etc.
That said, it's still extremely easy get embrace a culture were unexplainable errors are ignored. Especially in a cloud environment.
[1]: https://en.wikipedia.org/wiki/Ariane_5#Notable_launches
> It's the end result of a process
In my experience, it's even more than that. It's a culture.
They don't need to be especially talented engineers, but, in my experience (and I actually have quite a bit of it, in this area), they need to be dedicated to a culture of Quality.
And it is entirely possible for very talented engineers to produce shite. I've seen exactly that.
That’s in fact the thesis for the entire Deming management philosophy, and in line with what I’m saying (you can produce high quality with a good process or a good culture, you don’t necessarily need high caliber individuals)
In my case, the company produced absolutely top-shelf stuff, but even relatively mediocre companies did well, using Deming’s techniques. It required that everyone be on board, wrt the culture, though.
But I have found that a “good” engineer is one that takes their vocation seriously. They may not be that accomplished or skilled, but they have self-discipline, humility, and structure.
I’ve met quite a few highly-skilled “not-good” engineers, in my day. I’m embarrassed to say that I’ve hired some of them.
[citation needed]
> software quality doesn't appear because you have good developers
Good developers are a necessary ingredient of a much larger recipe.
People think that a good process means you can toss in crap developers, or that great developers mean that you can have a bad process.
In my experience, I worked for a 100-year-old Japanese engineering company that had a decades-long culture of Quality. People stayed at that company for their entire career, and most of them were top-shelf people. They had entire business units, dedicated to process improvement and QA.
It was a combination of good talent, good process, and good culture. If any one of them sucks, so does the product.
No one is working on quality, everyone works on new features. There is usually no incentive to increase quality, to improve speed, performance, etc.
If this industry wants to be respected, it should start trying to be actual engineers. There should be tons and tons of standards which are enforced legally, but this is not often the case. Imagine if there were no real legal guardrails in, say, bridge building!
edit: and imagine if any time you brought up this issue, bridge builders cockily responded with "well stuff seems to work fine so..."
When it comes to safety stuff (like bridge building), there are (and should be) strict licensing requirements. I would have no problem requiring such for work on things like medical equipment, We already require security clearance for things like defense information (unless you're a DOGE bro, I guess). That's a bit different from engineering creds, but it's an example of imposed structure.
But I think that it would be ridiculous to require it for someone that writes a fart app (unless it's a weaponized fart app).
What is in those requirements then becomes a hot potato. There are folks that would insist that any "engineer" be required to know how to use a slide rule, and would ignore modern constructs like LLMs and ML.
I'm not kidding. I know people exactly like that. If they get authority, watch out. They'll only "approve" stuff that they are good at.
On the other hand, if the requirements are LeetCode, then it's useless. A lot of very unqualified people would easily pass, and wreak havoc.
From what I can see, the IEEE seems to have a fairly good grasp on mixing classic structure and current tech. There's some very good people, there, and they are used to working in a structured manner.
But software has developed a YOLO culture. People are used to having almost no structure, and they flit between organizations so rapidly, that it's almost impossible to keep track of who is working on what.
The entire engineering culture needs to be changed. I don't see that being something that will come easily.
I'm big on Structure and Discipline. A lot of it has to do with almost 27 years at a corporation with so much structure that a lot of folks here, would be whimpering under their standing desks.
That structure was required, in order to develop equipment of the Quality they are famous for, but would be total overkill for a lot of stuff.
I do think that we need to impose structure on software supply chains, though. That's not something that will be a popular stance.
Structure is also not cheap. Someone needs to pay for it, and that's when you become a real skunk at the picnic.
IMO this should be the standard - software engineer should be a protected title, and everyone else would be titled some flavour of software developer or similar.
Now I'm not an engineer nor at all aware of what these standards actually mean, I'm sure they're pretty common sense and nowhere near as detailed as bridge building standards.
This, exactly this! I am a retired Aerospace engineer. 8 years of engineering studies at college and 2 years of work study before I was hired.
My son considers himself a "software engineer" and I have told him many times that he is NOT an engineer. He was homeschooled (so, pat my back there), never went to college, never studied programming at all. Yet he makes between $200K-$250K per year, 5 TIMES what I made as a Senior Engineer at Boeing. smh
I have years of experience at Boeing designing aircraft parts. The guiding principle is that no single failure should cause an accident.
The way to accomplish this is not "write quality software", nor is it "test the software thoroughly". The idea is "assume the software does the worst possible thing. Then make sure that there's an independent system that will prevent that worst case."
For the Therac-25, that means a detector of the amount of radiation being generated, which will cut it off if it exceeds a safe value. I'd also add that the radiation generator be physically incapable of generating excessive radiation.
One of the biggest things I see in junior engineers that I mentor (working in backend high throughput, low latency, distributed systems) is not working out all of the various failure modes your system will likely encounter.
Network partitions, primary database outage, caching layer outage, increased latency ... all of these things can throw a spanner in the works, but until you've experienced them (or had a strong mentor guide you) it's all abstract and difficult to see when the happy path is right there.
I've recently entirely re-architected a critical component, and part of this was defense in depth. Stuff is going to go wrong, so having a second or even third line of defense is important.
But same with code itself, a junior will have code that is "theirs", a medior/senior will (likely) work at scales where they can't keep it all in their heads. And that's when all the software development best practices come into play.
I agree with the previous poster that the feedback from the field is lacking a lot. A lot of doctors don’t report problems back because they are used to bad interfaces. And then the feedback gets filtered through several layers of sales reps and product management. So a lot of info gets lost and fixes that could be simple won’t get done.
In general when you work in medical you are so overwhelmed by documentation and regulation that there isn’t much time left to do proper engineering. The FDA mostly looks at documentation done right and less at product done right.
I'm a product manager for an old (and if I'm being honest somewhat crusty) system of software, the software is buggy, all of it is, but its also self healing and resilient, so while yes, it fails with somewhat alarming regularity, with very lots and lots concerning looking error messages in the logs, but it never causes an outage because it self heals.
Good systems design isn't making bug free software or a bug free system, but rather a system where a total outage requires N+1 (maybe even N+N) things to fail before the end user notices. Failures should be driven by at most, edge cases - basically where the system is being operated outside of its design parameters, and those parameters need to reflect the real world and be known by most stakeholders in the system.
My gripe with software engineers sometimes, they're often too divorced from real users and real use cases, and too devoted to the written spec over what their users actually need to do with the software - I've seen some very elegant (and on paper, well designed) systems fall apart because if simple things like intermittent packet jitter, or latency swings (say between 10ms and 70ms) - these are real world conditions, often encountered by real world systems, but these spec driven systems fall apart once confronted with reality.
I agree. In my current job, no developer ever talks to users. Projects involve multiple layers of subcontractors, and cross-vendor APIs are part of the bid/spec. Each layer’s purchasing department is in perpetual Bidding Mode while dev teams are already in the middle of doing the actual work. So by default, everyone is Strongly Discouraged From Talking To $VENDOR | $SUBCONTRACTOR | $USER.
I will say that me pretending to know how to best design medical equipment as a web developer is pretty full of myself haha. Highly doubt whatever I'm spouting is a new idea. The idea of working on this sort of high-reliability + high-recoverability systems seems really interesting though!
The Therac-25 was meant to have a detector of radiation levels to cut things off if a safe value was exceeded, but it didn't work. It could obviously have been improved, but you always have the possibility that "what if our check doesn't work?".
In the case of the Therac-25, if the first initial failures had been reported and investigated, my understanding is (I should make clear I'm not an expert here) it would have made the issues apparent, and it could have been recalled before any of the fatal incidents happened.
In a swiss cheese model of risk, you always want as many layers as possible, so your point about a detector fits in there, but the final layer should always be if an incident does happen, and something gets past all our checks, how can we make it likely that it gets investigated fully by the right person.
But this opens up a can of worms, as suddenly you have to deal with every edge case, test for every possible input, etc. This was before fuzz testing, too. Each line of defensive coding, every carefully crafted comment, etc all added to the maintenance burden; I'd even go as far as claim it increased uncertainty, because what if I forgot something?
15 years later and it feels like I'm doing far less advanced stuff (although in hindsight what I did then wasn't all that, but I made it advanced). One issue came up recently; a generic button component would render really tall if no label was given, which happened when a CMS editor did not fill in a label in an attempt to hide it. The knee-jerk response would be to add a check that disallows empty labels, or to not render the button if no label is given, or to use a default button label.
But now I think I'll look at the rendering bug and just... leave the rest. A button with an empty label isn't catastrophic. Writing rules for every possible edge case (empty label, whitespaces, UTF-8 characters escaping the bounds, too long text, too short text, non-text, the list goes on) just adds maintenance and complexity. And it's just a button.
It's not that great developers aren't necessary for software quality, more that they aren't sufficient.
Good product cultures are ones where natural communication between the field and engineering would mean issues get reported back up and make their way to the right people. No process will compensate for people not giving a shit.
I think the bit I quoted, especially if you read in the context of the article, is talking about culture. I.e. it's talking about a process that informs software development, management and sales. Things like formal proofing and type systems are the exact kind of processes that aren't what it's talking about.
I kind of agree with you though about the process/culture distinction - ultimately, if you don't have a culture where people actively care about improving reliability, any process is just gonna become a tick-box exercise to appease management.
If you only take one thing away from this article, it should be this one! The Therac-25 incident is a horrifying and important part of software history, it's really easy to think type-systems, unit-testing and defensive-coding can solve all software problems. They definitely can help a lot, but the real failure in the story of the Therac-25 from my understanding, is that it took far too long for incidents to be reported, investigated and fixed.
There was a great Cautionary Tales podcast about the device recently[0], one thing mentioned was that, even aside from the catasrophic accidents, Therac-25 machines were routinely seen by users to show unexplained errors, but these issues never made it to the desk of someone who might fix it.
[0] https://timharford.com/2025/07/cautionary-tales-captain-kirk...