There is a huge swathe of problems where AI can provide tons of value, and where most of the benefits and costs of getting it wrong lies with the person deciding to trust the AI in the first place.

Your AI self driving car doesn't need to explain itself, it just needs to arrive safely. An AI that merely supports a human decision maker also might not, depending on how its applied. But the AI that might deny your access to treatment for a disease, reject your mortgage application or recommend to a patrolling police officer that he should pull you over, and not the next car -- those AIs should explain themselves.

If a self driving car decides to ram into a tree/another car at full speed regularly, im guessing there will be a huge demand for explainability, not just a collective shrug and "I guess we need to train it more dunno lol".

I think it could be useful to distinguish systems which are required to automatically explain every decision they recommend as a matter of routine, versus systems which are not, but still are required to support post-mortem investigation of serious failures.

Yes, but at least the person bearing the cost is the same that decided to let the AI drive. That doesn't mean what you describe is desirable, but the fall out is rather contained to people who have consented to using the AI. Not so with the examples I listed.

but you are wrong, there are people who did not consent to using AI being injured in those situations, same as the police/sentencing example you brought up

What if the AI drove the car into a pedestrian? The pedestrian didn't consent to letting the AI drive.

It's a spectrum, not black and white. The self driving car is a lot closer to one end, and the other examples are closer to the other. None are perfect examples of the extremities.

What if a drunk drives the car into a pedestrian? Not much content there, either.

> What if a drunk drives the car into a pedestrian?

This is easy to explain. The driver is at fault. Explainability is what we are debating.

So the AI is at fault then.

the question becomes what can you >do< about it if you have no understanding of the logic it operates on.

Very much not.

Becauae there’s nothing to explain. The software is broken and needs to be fixed, and if the problem is severe and the manufacturer knowingly tries to cover it up, that manufacturer is culpable of negligence.

The software has no more agency than a cruise control.

Good luck explaining that to a judge who demands to know why the AI made the decisions that caused an accident, and what exactly the manufacturer is doing to prevent it from happening again.

There are tons of physical processes that aren't explainable. Most current petrochemical plants have inner workings that are only half known, at best. What are we doing about it? Modeling the failure, and engineering a solution that no longer fails in that situation. Unit testing our way to robustness!

> Most current petrochemical plants have inner workings that are only half known, at best. What are we doing about it?

When these processes kill people we very much do somethint about them.

> Becauae there’s nothing to explain. The software is broken and needs to be fixed

The second statement kinda denies the first. If broken software causes a severe or expensive problem, people generally expect an explanation. That is what a post-mortem is generally for.

>> ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it.

I have to be able to debug anything I work with; debugging is necessary.

>> "the neural network parsing the LIDAR did not classify the stop sign correctly, so the driving algorithm did not stop"

It's important to look into why the stop sign was mis-classified, even if the classifier is a subcomponent.

I would go further: AI in a system that is in charge of safety critical system needs to be understood perfectly by its engineers. Right now, the community is trying to get by with shrugging and saying "dunno, but look at the reliability". In the long run this is not going to be good enough. We will need tools to dissect trained systems and build complete explanations of why it works (or doesn't).

It seems to me that understanding these deep learning models will be very tough since the reason why ml was necessary in the first place was the sheer complexity of the solution being too much for generations of programmers to solve

Understanding an existing solution is very different from trying to find it. We get taught things at school that seem simple and obvious to us now, but were extremely hard to discover or invent.

I see these training approaches as a tool that makes machines search the space of potential solutions for certain types of problems more efficient than humans. The next logical step is to try and understand these solutions and improve upon them.

Consider adversarial input. Right now we cannot tell which classes of adverserial inputs can exist for a given NN. We can only try to find representative examples. If you had a good enough understanding of how the NN works internally, you have a chance to derive the full set of adversarial inputs or - maybe -prove their absence.

I think part of the problem here is what 'explainable' means and to whom. A Deep learning or Data Analyst may be looking for or accept one kind of answer (or even be able to comprehend it), while a Medical Doctor is looking for something else to hang his hat on, and the layman has an even different (lower?) expectation of explanaition.

Now perhaps we could design another set of algorithms (NN generated?) that would present the right type of explanation to each. Frankly, I'm not very interested in the layman's explanation, because what they want to hear is so disconnected from what's happening that it often seems irrelevant. That said, the placebo effect is real so you want the doctor and patient convinced that they are doing the right thing... so maybe it's more important (edit to add) for actual medical outcomes?

>But if the non-explainable system is better even than experts, I think most people would want the non-explainable system.

Maybe? You'd have to convince me in areas as medicine. Why are you recommending this treatment path? "Dunno. The computers says so."

Mind you, the doctor may well not in reality have a lot more solid basis for his recommendation. But at least there's the appearance of logical justification.

Is that because you care about verifiability though, or actually interpretability? If the system provides you a diagnosis without context, but later can be confirmed by actually testing for that condition, do you still refuse treatment because you don't know how the original diagnosis was suggested?

Oh, if it's a recommendation for running some tests, I can't imagine much pushback unless they're particularly invasive or expensive. It's not like doctors don't order large panels of tests at the drop of a hat.

If you then have treatment based on those tests, that's really a doctor being assisted by an AI at that point.

The current explanation for anything in medicine is "Dunno, we tested it and it seems to work". It is a much better one than "The computer says so", but it's one the computer can tell you too.

> explainability is necessary to use it.

For that matter, freedom from daily blue screens and reboots is not needed to use an operating system.

All sorts of non-AI software has confusing states and quirks, yet is in wide use.

The poll is biased. There is no evidence that performance and explainability are at odds.

With our current models they definitely are at odds. There is a strong correlation between higher average performance and lower explainability when comparing machine learning approaches like neural nets, linear models, decision trees, etc for nearly all tasks.

I cannot explain how I form the sentences I speak. I still find speaking useful.