You are mixing two things here, general purpose AI/ML applications are indeed hard to trust and test since you don't have any specifications to test against, there are some explicit and implicit benchmarks but nothing is guaranteed to cover the whole (well, most of the) spectrum of inputs and outputs.

On the other hand AI's used for specific applications and purposes can be tested thoroughly, you know that FSD sucks because it was tested and shown not to do it's intended work properly. You can test a medical imaging interpretation AI by comparing the results to a human doctor, and you can test test an in house climate system controlled by AI by judging its performance.

Nope. My point is in general with AI with deep neural networks as black boxes being very untrustworthy no matter the application, unless it is for non-serious uses.

Using deep learning AI in medical imaging is just as opaque to a doctor as rolling a dice as a random number generator to see if won't get confused by a faulty pixel or bad input and misinterpret what it is seeing for something else, leading to mis-diagnosis, which is a life changing risk.

It is no good for them to use the output of an AI that just 'explains' by giving a number of 99% and its incorrect without any transparent way of reasoning for that output or explanation which is exactly what medical professionals want to avoid, hence multiple doctors have to review it because they *do not* trust it.

Thanks for the response. So is it more of a feeling of trust for the end-users that people want explainable AI? If an explainable model has the same performance as a black box model, does that really add any real world benefit though?

Or does that explainability potentially offer something to the FSD or fintech engineers/developers that allows them to make better models? And if so, what would that be? That’s where I seem to get stuck on this question, explainability sounds nice but I want to understand what specific, practical it would offer.