I don't really agree with you.

> If you were rejected by a bank for a home loan, you would want to know why your creditworthiness wasn’t evaluated positively (and banks must explain precisely why to regulators).

This is why we don't use humans to evaluate credits, but precise algorithms. Humans are just applying the algorithms, they are not evaluating themselves with their gut feeling. I don't see why this would change with AI. Regulations prevents unexplainable tools to be used there, so deep learning black box models will not be used, similarly to why humans are not used today.

But in cases where performance is required, but not explainability, then deep learning will strive. And I believe that cases where explanation is required is only a very small subset of areas where AI could be useful.

> And if a self-driving car made a decision in a collision

Would you prefer a car that crashes once every 100 million miles but is not explainable, or a car that crashes once every 100 miles but is interpretable can explain why it crashed ?

That's ok, but based on your example, how would a human apply an algorithm that it can't determine why it worked?

We're not talking about "Not Hotdog" here. Application of a model in a real-world at-scale scenario is a lot more than running inference and walking away.

At a bank credit decisions are evaluated by humans, frequently and often. These reviews are conducted in the forms of sampling audits, control processes, and other scenarios that involve internal bank employees and external regulators. In each case, humans will inspect the details of what occurred. This would be impossible with any type black-box model (SSL, deep NN, etc.).

I'm not sure I understand your comment.

Bank credits are auditable, because there are some precise algorithm around them. Humans created the algorithm specifically so that it is interpretable. Then they must apply it the same way to everybody. They can't just say "oh, I think this person is more likely to actually reimburse, I play golf with him and I trust him !"

This is a specific case where no black box will ever be used (at least I hope).

There's incentive to use complex black boxes everywhere, if more profitable.

Of course, and my point is that it's perfectly fine, and even desirable, as better models lead to better outcomes.

Of course, for cases where explainability is needed, either required by law (such as banking), or by common sense, then black boxes will not be deployed.

Isn't it interesting that an industry like banking, built on those beautiful, precise algorithms, blows up so regularly?

You're really thinking of finance. Pure banking is solved long ago.

We want our models to be opaque in the same way we like to ask people why they made a certain decision or act in the way they do. What's interesting to me is if you want to know about a person you'll get better information by asking their closest friends than themselves.

Perhaps there's something in the black box nature of our own self understanding similar to these hyper complex function approximators.

Judges make harsher rulings when they're hungry. A regulator might decline a loan and justify it b/c they had a bad experience with someone that reminds them of the person they're dealing with. AI makes bad decisions in boundary conditions or when they're trained on biased data. I think it's good to strive for opacity in every case but in complex decision spaces I'm not sure if we'll ever get fully satisfactory explanations.

Which is a meandering way of making the non-point that models making judgments have made me re-evaluate what I trust and why, and I think in many cases I'd rather trust a black box if I know what's gone into it and what's come out.

> Judges make harsher rulings when they're hungry.

Actually, that finding is highly improbable: https://nautil.us/blog/impossibly-hungry-judges

Everywhere you wrote opaque you meant transparent.

High risk models should work with humans in the loop, not autonomous.

Yes, this is the point I am trying to make. Self-supervised models using already opaque techniques built by an ethically flexible company is not a good recipe.

A sufficiently complex algo black box could fool the people.

Would this rule out self driving cars from the get go?

Depends on what you mean by self driving cars. To me it is not even clear that driving assist tech will ever be advanced enough for cars to drive themselves reliably.* But that aside. Human in control + driving assists is already safer than fully manual driving. The question is whether the opposite is even safer. It is obvious to me that self driving with the human as backup is a joke. Humans cannot react fast enough in real emergencies. So if you want the car to mostly drive itself, it has to actually always drive itself. And to me it is not clear whether this is really going to be safer than the human+driving_assist scenario.

*Of course I am talking about tech based on the current DL approaches. If we had AGI, then the question does not even need to be asked.