Yeah - I work a lot in healthcare associated infections, and a good 80% of the stuff I read about "We're from Silicon Valley and we're here to help..." just ends with me going "They're going to kill people."

I've second hand experience with this. Though not a lethal situation (thank the heavens!) the mindset was dead on. A grad student in the BioEng lab was doing her PhD on a fall detection sensor for the elderly. Kinda like what the new Apple watch is supposed to do, but, you know, something that actually works. The lab was working with a device manufacturer that was looking to branch into medical devices as a new market and was partnered with the lab to do the initial PoC. Well, one day she came into the lab to discover that the company had shut down and remote bricked the devices overnight. They were still in early testing, so no people were harmed, which was really lucky. But if they had been under clinical trial already, there was a real possibility that real people out there would have been harmed severely (a 'bad' geriatric fall can become lethal).

All emails to the company were unanswered. All phone calls went to voicemail. Nothing could be done. She was 5 years into her PhD on this and had to restart the whole project (she wasn't much of a coder, more a clinical trials person).

In medicine, the SV mindset of 'move fast, break things' means that granny is breaking her hip and isn't going to last much longer. You CANNOT 'break things' in medicine. However, if you do get though clinical trials, hoooo baby! You essentially have a monopoly and will be rolling in it. MedTech investing is a lot like SV's VC investing, just a lot more careful.

What most people dont realize, depending on the country, those press for emergency buttons for the elderly are not considered an emergency call. Including the advertisement you see on buses targeting people fearing a heart attack. Meaning, it is not guaranteed that the message will go through. It is an app like any else with the same reliability. That was fortunately the point where my former boss declined a contract.

> Kinda like what the new Apple watch is supposed to do, but, you know, something that actually works.

Does the Apple watch not work?

> But if they had been under clinical trial already, there was a real possibility that real people out there would have been harmed severely (a 'bad' geriatric fall can become lethal).

How would that be different if they didn't have a device? They would have died anyway, right?

> How would that be different if they didn't have a device? They would have died anyway, right?

Maybe. If they think that the device will help them in their hour of need, then they may be taking chances they normally would not have. To have something remote brick and then not tell the people that it was bricked is a gigantic ethical violation.

> Does the Apple watch not work?

Not to a medically relevant level:

"Apple Watch cannot detect all falls. The more physically active you are, the more likely you are to trigger Fall Detection due to high impact activity that can appear to be a fall.”

https://support.apple.com/en-us/HT208944

> Apple Watch cannot detect all falls…

If we pick apart disclaimers, I think we will find that nothing works. /gentle_sarcasm

I think this is something that isnt that present in our profession as it should be. If you are an electrical engineer, it is clear and present, that your products might kill someone if you screw up. The same goes for a mechanical engineer. And it will be your responsibility. If you screw up and should have known better, you might even go to jail for negligence.

We on the other hand have the Therac-25.

We claim to be on a level of engineering fields. We should also act that way.

Agreed. The Order of the Engineer is a good place to start and bringing that tradition down into comp-eng/sci would be a nice start.

https://en.wikipedia.org/wiki/Order_of_the_Engineer

Interesting, had not heard of that. What should we computer programmers, we flexible always trying new things without enough tried and true practices think of real engineers, what should they think of us?

Real engineers should probably look down on us from a great height.

Here's a more humorous take: https://imgs.xkcd.com/comics/voting_software.png

Therac-25 was thirty years ago, and still is the go to example. That made me wonder. If you compare, for example, https://en.wikipedia.org/wiki/List_of_bridge_failures#2000–p... or https://en.wikipedia.org/wiki/List_of_structural_failures_an... with https://en.wikipedia.org/wiki/List_of_software_bugs, software doesn’t look that deadly.

People generally don't attribute failures to software bugs if there is anything else that contributes to failure. The list in particular is missing the loads of accidents caused by bad UI; incidents of "I was just following the GPS's/autopilot's directions" that result in death are missing (such as the KAL 007 airliner incident).

The Therac-25 is the go-to example because it is the best documented and least-contested case. Also, software usually isn't in ultimate control, most systems have additional (mechanical) safety controls.

Mars Climate Orbiter, 1999: http://articles.latimes.com/1999/oct/01/news/mn-17288. Undetected metric conversion error (though unclear if this was a manual process or software).

Toyota unintended acceleration, 2010: https://www.edmunds.com/car-safety/for-toyota-owners-uninten.... Toyota maintains that it was not caused by software error.

Now let's list some cases where the software did have ultimate control:

Schiaparelli EDM lander, 2016: https://newatlas.com/esa-schiaparelli-mars-crash-inquiry/496.... Faulty decision-making in the automated descent system due to input saturation.

Tesla auto-pilot crash, 2016: https://www.theguardian.com/technology/2016/jun/30/tesla-aut.... Faulty decision-making due to incorrect image analysis. In this case, the additional safety controls (the driver himself) failed too.

Uber car crash, 2018: https://money.cnn.com/2018/03/20/news/companies/self-driving.... Faulty decision-making due to incorrect image analysis, though the investigation is still ongoing, so no definite cause determined (FAFAIK).

Software doesn't have many opportunities to be deadly, and when said opportunities do exist, this is frequently in regulated fields where safeties are mandated by law (interlocks on a laser or a welding robot arm etc).

I'd imagine a lot of places where software is part of a safety critical system also come with physical fail safes. An example being burst plates on pressure vessels, if the software that controls pressurization spasms and causes an over pressure the physical fail safe will prevent a catastrophic accident.

Software engineering is more like witch-doctoring than to civil engineering. It is not remotely price-efficient to prove completeness on programs.

If you expect your doctor's software to be perfect, be prepared to pay for 2 software engineers salaries per doctor.

I agree that we act recklessly sometimes, but I hope I'm not alone among software engineers in believing that this finding from the committee was just plain stupidity which should not be done by any engineer worth his or her salt:

"AECL had never tested the Therac-25 with the combination of software and hardware until it was assembled at the hospital."