You are a black box AI. I can nevertheless trust you to classify dogs vs cats.
We're talking about algorithms that are based on a simplified neural architecture, no redundancy, no self reflection and are still quite immature.
Nevertheless we're being asked to trust a black box AI, that you cannot interrogate?
Yes, of course, what's the worst that can happen?
On the other hand, if you place Magnus Carlsen against AlphaZero in a game of chess, I will bet on AlphaZero. If however you reduce the complexity of AlphaZero down to a level that it can produce an explanation I can understand, I would instead bet on Magnus Carlsen.
Of course we should care about the quality of AI systems, but chasing a human understandable explanation is just the wrong way to go about it, since it in many cases necessarily limits quality of the decisions.
You don’t need to reduce complexity to induce explainability. You just need to decompose the function into smaller parts which you can understand.
Contrastive LRP for example is a Function decomposition technique for explaining deep neural networks with high fidelity.
There are a whole set of mathematical functions that are beyond the range of a typical human mind to understand. But there are proofs that exist that explain how they are correct.
I don't think it's a big ask for a new model to be able to justify its own determinations.
Of course, that makes things slightly more difficult for all those in research and those selling snake oil and everyone in between. But that's what the difference between science and alchemy is isn't it?