Usually in medical design for class III devices you have to either have some form of hard interlock or an additional redundancy, like a standalone microcontroller for interlocking or a voting system. Hazard Analysis is definitely a thing that is very finely combed by regulators and definitely something companies don't want to fall short on.

I went with a mechanic in a chair lift once. He was very nervous, saying he knows the kind of people who maintain these things, and he knows the kinds of things that can go wrong. I feel the same way about automation, whether it be self driving cars or an X-ray machine. I know the kind of people that program them, and the vast number of ways things can go wrong.

How many CV engineers own a self driving car becomes the AI equivalent question?

I work in the medical device field. Funny you should say that because just yesterday I was looking at some really crappy (but still safe, mind you!) code and wondering how I'd feel if I were being wheeled into the OR, knowing "how the sausage is made."

Do we at least still use hardware interlocks? I hope we still use hardware interlocks.

Hardware "interlocks" are used when the risk cannot be reduced to an acceptable level by software alone.

With self-driving cars, you also need to consider the kind of people who might be driving them manually. Drunk, sleepy, prone to road rage, distracted by noisy kids in the car, thinking about their financial or relationship problems... there's a terrible amount of death and shattered lives to consider there.

If I remember right, during the height of the most recent Iraq war, a U.S. soldier had a higher probability of being killed in a traffic accident while in the United States than being killed while actively deployed in Iraq.

If a (so-called) self-driving car screws up and kills some people, it's headline news everywhere. If Joe Blow downs five shots of scotch after being chewed out by his boss and crashes into a carload of kids, it barely makes the local news.

>he knows the kinds of things that can go wrong

Sounds like he didn't know nearly as much as he thought. Chairlifts are remarkably safe.

https://newtoski.com/are-chairlifts-safe/

I get the point you're making though.