Unfortunately any AI systems will encounter the verification dilemma, the more powerful an AI system becomes, the less verifiable it would be.

Verification is easy, just like with humans. It's called a driver's license. And since the AI does not tire and can probably be sped up, you can put it through hundreds of thousands of hours of driving quickly in a good simulator with adversarial and normal situations, then rate it at various tasks. Just like we should do with human drivers, but fail to.

Explanation is harder. But we probably shouldn't care, even in courts people cannot often explain what and why they did while driving, or they just lie. The thing is, for liability purposes you have to ensure it is not a series defect and that a good human driver would not be able to handle it

In the world of investments, that would be called buzzwordily as “back testing”.

But is it sufficient to answer _why_ the machine chose to act a certain way given a certain set of instantaneous input criteria?

No. But, as the parent post said, we don't always know why a human driver chose to act a certain way either. So that shouldn't be a blocker.

But we evolved to have empathy to help understand how humans act in the cases we don't have complete or good information.

This is why "crazy" people make us so uneasy. They don't fit our mental models for how a human should act. Would you be comfortable driving with a road full of unpredictable "crazies"?

I wonder if we'll ever have the same level of trust with AI as humans if it is still being used at a black box level.

Bad AI will suffer from the Dunning-Kruger effect and overestimate its abilities, while good AI will suffer from Imposter Syndrome and underestimate its abilities.