Good luck explaining that to a judge who demands to know why the AI made the decisions that caused an accident, and what exactly the manufacturer is doing to prevent it from happening again.

There are tons of physical processes that aren't explainable. Most current petrochemical plants have inner workings that are only half known, at best. What are we doing about it? Modeling the failure, and engineering a solution that no longer fails in that situation. Unit testing our way to robustness!

> Most current petrochemical plants have inner workings that are only half known, at best. What are we doing about it?

When these processes kill people we very much do somethint about them.

> Becauae there’s nothing to explain. The software is broken and needs to be fixed

The second statement kinda denies the first. If broken software causes a severe or expensive problem, people generally expect an explanation. That is what a post-mortem is generally for.