The underlying reason why high confidence is not enough is that even strong/confident correlations could be misleading when seen in causal light — a black box model trained to predict credit performance might be very confident in rejecting loans for applicants from “poorer” zip codes and approving those from “richer” zip codes — even though those are not actual causes... therefore somebody could exploit the system by renting an address in a rich neighborhood for a couple of months when taking out a big loan (the analogue of adversarial examples).
Your example points to models that provide low quality uncertainty estimates, but that's not true for all deep learning models.
I believe it's these low quality uncertainty estimates that lead people to look toward "explainability" as a solution, but for the majority of use cases, I think people just want better uncertainty estimates so that they can "know when they're model doesn't know".
There are techniques now to get higher quality, calibrated, uncertainty estimates that don't suffer from the problems you mentioned and I've outlined these solutions in my posted link above.
Additionally if you're interested, there is some nice recent research from google on the subject:
https://ai.googleblog.com/2020/01/can-you-trust-your-models-...
and from oxford:
> calibrated and robust uncertainty estimates NOT low quality and uncalibrated uncertainties estimates.
Could you explain what you mean by “calibrated” and briefly summarize the essential idea behind what allows the learning of robust uncertainty estimates, if not a causal understanding?
If you haven’t already, look up work by Scholkopf, Janzing , Peters and co (over the last decade) for a justification of why causal reasoning is exactly what you want if you want to generalize across covariant earth/dataset shift (which is basically what the Google blog post is about).
Considering that ML has access to orders-of-magnitude greater quantities and resolution of information that we do, why do we expect their decision structure to be intensely linked to our own decision process.
My counterpoint is when a model trained on dog breeds misidentifies a breed with a high certainty factor, the model should be able to justify (explain) why it made that judgement.
When people ask for explainable models, what they really want (in my opinion) is calibrated and robust uncertainty estimates .
Good uncertainty estimates would let them know when to trust a model's prediction and when to ignore it.
For example, a model trained to predict dog breeds should know nothing about cat breeds, and there should be some way to quantify when it doesn't know!
I've been doing a review of techniques that are becoming more popular in this area:
https://blog.everyhue.me/posts/why-uncertainty-matters/