In the world of investments, that would be called buzzwordily as “back testing”.
But is it sufficient to answer _why_ the machine chose to act a certain way given a certain set of instantaneous input criteria?
No. But, as the parent post said, we don't always know why a human driver chose to act a certain way either. So that shouldn't be a blocker.
But we evolved to have empathy to help understand how humans act in the cases we don't have complete or good information.
This is why "crazy" people make us so uneasy. They don't fit our mental models for how a human should act. Would you be comfortable driving with a road full of unpredictable "crazies"?
I wonder if we'll ever have the same level of trust with AI as humans if it is still being used at a black box level.
Explanation is harder. But we probably shouldn't care, even in courts people cannot often explain what and why they did while driving, or they just lie. The thing is, for liability purposes you have to ensure it is not a series defect and that a good human driver would not be able to handle it