We're not talking about "Not Hotdog" here. Application of a model in a real-world at-scale scenario is a lot more than running inference and walking away.
At a bank credit decisions are evaluated by humans, frequently and often. These reviews are conducted in the forms of sampling audits, control processes, and other scenarios that involve internal bank employees and external regulators. In each case, humans will inspect the details of what occurred. This would be impossible with any type black-box model (SSL, deep NN, etc.).
Bank credits are auditable, because there are some precise algorithm around them. Humans created the algorithm specifically so that it is interpretable. Then they must apply it the same way to everybody. They can't just say "oh, I think this person is more likely to actually reimburse, I play golf with him and I trust him !"
This is a specific case where no black box will ever be used (at least I hope).
Of course, for cases where explainability is needed, either required by law (such as banking), or by common sense, then black boxes will not be deployed.
> If you were rejected by a bank for a home loan, you would want to know why your creditworthiness wasn’t evaluated positively (and banks must explain precisely why to regulators).
This is why we don't use humans to evaluate credits, but precise algorithms. Humans are just applying the algorithms, they are not evaluating themselves with their gut feeling. I don't see why this would change with AI. Regulations prevents unexplainable tools to be used there, so deep learning black box models will not be used, similarly to why humans are not used today.
But in cases where performance is required, but not explainability, then deep learning will strive. And I believe that cases where explanation is required is only a very small subset of areas where AI could be useful.
> And if a self-driving car made a decision in a collision
Would you prefer a car that crashes once every 100 million miles but is not explainable, or a car that crashes once every 100 miles but is interpretable can explain why it crashed ?