Human intelligence isn't explainable, though. We can make up explanations but they're not based on our actual neural function (heck, we don't even know how that works in general, let alone how we generate specific outcomes).
A large part of human success is that we are able to transmit knowledge to each other and accumulate experience over generations. A human starting from scratch is not very much more than its ape cousins.
Human beings can explain efficiently because we must learn concepts decomposably and can take deductive shortcuts. In its purest form we have mathematical proofs as explanations. Explanations work with core concepts and generates on that basis. E.g. We want to know A and we know B, B is applicable in condition C. We will show condition C applies. With B, A can be shown in terms of...
When you give big data to machines they work with raw correlations and output predictions. But humans have to be more creative because our scratch space is so small. Newton took Brahe's big data and compressed it to a theory of universal gravitation. Darwin made a relatively small number of observations, all things considered, and came up with On the Origin of Species.
I am still not sure but after seeing Schmidhuber's talk last week, I think it is being able to find regularities/patterns, resulting in a compressed representation that allows to understand and compress some aspect of the world, and building layers (abstractions) of those compressed representations. This allows you to navigate through very large search spaces much much faster.
It's a fascinating subject, and at the core of what we are. I recommend the film.
https://en.wikipedia.org/wiki/The_Man_Who_Knew_Infinity_(fil...
You don't need to explain every dendritic cluster in order to extract core, composable and generative abstractions useful enough to shorten time to understanding by orders of magnitude (discovery of zero vs how quickly it can be taught).
We know a lot more about how machine learning works--since we developed it ourselves recently--but because it is so new, we feel the need for deeper understanding to mitigate risks. To go back to the suicide example, a computer-driven system will destroy itself immediately for all sorts of reasons, including incredibly trivial errors.
To gain intuition and reasoning of a model is to have understanding and trust--transparency. When you strike a nail with a hammer, it's pretty predictable what might happen: the nail could get hit, the hammer could miss, or very rarely, the hammer's head may fly off of the handle. When you replace the hammer with a black box that works correctly 99.999% of the time, but for 0.001%, something completely unpredictable happens, then there's a problem with volatility because that unpredictable event may have unacceptable consequences. I think explainable AI could help with intuitive and more fine-grained risk analysis, and that's certainly a good thing in high-stakes applications such as defense.
[0] ISLR, Page 25: http://www-bcf.usc.edu/~gareth/ISL/ISLR%20First%20Printing.p...