I think it could be useful to distinguish systems which are required to automatically explain every decision they recommend as a matter of routine, versus systems which are not, but still are required to support post-mortem investigation of serious failures.

Yes, but at least the person bearing the cost is the same that decided to let the AI drive. That doesn't mean what you describe is desirable, but the fall out is rather contained to people who have consented to using the AI. Not so with the examples I listed.

but you are wrong, there are people who did not consent to using AI being injured in those situations, same as the police/sentencing example you brought up

What if the AI drove the car into a pedestrian? The pedestrian didn't consent to letting the AI drive.

It's a spectrum, not black and white. The self driving car is a lot closer to one end, and the other examples are closer to the other. None are perfect examples of the extremities.

What if a drunk drives the car into a pedestrian? Not much content there, either.

> What if a drunk drives the car into a pedestrian?

This is easy to explain. The driver is at fault. Explainability is what we are debating.

So the AI is at fault then.

the question becomes what can you >do< about it if you have no understanding of the logic it operates on.

Very much not.

Becauae there’s nothing to explain. The software is broken and needs to be fixed, and if the problem is severe and the manufacturer knowingly tries to cover it up, that manufacturer is culpable of negligence.

The software has no more agency than a cruise control.

Good luck explaining that to a judge who demands to know why the AI made the decisions that caused an accident, and what exactly the manufacturer is doing to prevent it from happening again.

There are tons of physical processes that aren't explainable. Most current petrochemical plants have inner workings that are only half known, at best. What are we doing about it? Modeling the failure, and engineering a solution that no longer fails in that situation. Unit testing our way to robustness!

> Most current petrochemical plants have inner workings that are only half known, at best. What are we doing about it?

When these processes kill people we very much do somethint about them.

> Becauae there’s nothing to explain. The software is broken and needs to be fixed

The second statement kinda denies the first. If broken software causes a severe or expensive problem, people generally expect an explanation. That is what a post-mortem is generally for.