As we start to use AI/ML for more tasks, the need for model interpretability rises. We expect doctors to explain their gut feelings, much like we expect computer vision models that detect disease to explain their findings and have a (theoretically sound) estimate of confidence.

SVM's were so popular, pretty much because they had a firm theoretical basis on which they were designed (or "cute math" as deep learners may call it). As Patrick Winston would ask his students (paraphrasing): "Did God really meant it this way, or did humans create it, because it was useful to them?". Except maybe for the LSTM, deep learning models are not God-given. We use them because, in practice, they beat other modeling techniques. Now we need to find the theoretical grounding to explain why they work so well, and allow for better model interpretability, so these models can more readily be deployed in health care and under regulation.

The article calls for an explanation not why some ML method works, but for an explanation of a particular ML result, like why a car drives this way or why a patient got a cancer. While I hopeful for the former, I just do not see the basis for the latter.

If some regulations shall require such explanation, the end result will be fake stories like parents tell to the children that Moon do not fall because it is nailed to the sky.

I don't think the paper asked for that. Relevant quote:

> machine learning is more concerned with making predictions, even if the prediction can not be explained very well (a.k.a. “a black­box prediction”)

So in your example: an algo may explain that a car slows down, before taking a turn, because else it would likely crash. It may even get to a threshold ("under these weather conditions, anything over 55Mph is unsafe when taking a turn of such and such degree"). Statistics can help with that.

Welling is not asking for deep learning models to explain why a person got a cancer, but to explain its reasoning when it diagnoses a person with cancer ("I am confident, because in a random population of a 1000 other patients, these variables are within ..."). Statistics can help with that. It aligns with their mind set and tool set.

Regulations are cheated even with these kind of explanations, but that is for another story (black box models may provide some plausible deniability).

I am referring to this fragment:

> Thus, for many applications, in order to successfully interact with humans, machines will need to explain their reasoning, including some quantification of confidence, to humans.

No doubtful there are cases when an explanation is easy. Often this is because we have a very solid model like physics of a car. In fact since we know the model, we do not need an explanation, we must demand that the algorithm follows the model or declare it unfit.

But how can we expect an explanation for a behavior in a critical situation on a road that was not explicitly programmed and when the algorithm decided to turn to a particular degree bases on a non-trivial inference? Similarly, when an algorithm decides if a patient needs an emergency operation or if they can wait, why can we expect an simple explanation especially for the patient with rare conditions when algorithm again must perform an inference, not a deduction from 1000 very similar cases?

Maybe you couldn't always expect an explanation, but having one would certainly be useful. Methods that could get the job done while also providing enough information to explain or at least shed some kind of light on why the solution was chosen would likely be preferable.

Just because you can't explain your gut feelings doesn't mean they are unexplainable, or that you don't have any obligation to explain them.

I hear what you're saying, but in terms of usefulness for business decisions, what leaders at a company would be satisfied with someone providing Vapnik's quote? Certainly machine learning and statistics have applications outside of business, but when it comes to realities in industry settings, very often an explanation of results is necessary, in addition to an explanation of why a particular machine learning approach is best for solving a given problem, how it works to solve the problem, etc.

The problem is not about how to take a scientific conclusion and make it suitably less scientific to serve as a business explanation.

The problem is to replace inept employees who believe "business decisions" are not scientific questions, so that over time there is a convergence to using the scientific method, with legitimate statistical rigor, when making a so-called business decision.

Generally speaking, the only people who want for there to be a distinction between a "business question" and a "scientific question" are people who can profit from the political manipulation that becomes possible once a question is decoupled from technological and statistical rigor. Once that decoupling happens, you can use almost anything as the basis of a decision, and you can secure blame insurance against almost any outcome.

This is why many of the experiments testing whether prediction markets, when used internally to a company, can force projects to be completed on time and under budget are generally met with extreme resistance from managers even when they are resounding successes.

The managers don't care if the projects are delivered on time or under budget. What they care about is being able to use political tools to argue for bonuses, create pockets of job security, backstab colleagues, block opposing coalitions within the firm. You can't do that stuff if everyone is expected to be scientific, so you have to introduce the arbitrary buzzword "business" into the mix, and start demanding nonsense stuff like "actionable insight" -- things that are intentionally not scientifically rigorous to ensure there is room for pliable political manipulation for self-serving and/or rent-seeking executives, all with plausible deniability that it's supposed to be "quantitative."

Edit: I wrote prior to this edit agreeing with alot of what you said. However, I mostly wanted to say that providing an explanation that people may be able to understand more easily doesn't necessarily make a scientific conclusion less scientific.

True. It's not necessary. But most times in real life when this happens, the scientific rigor is the first thing to go. This can manifest itself in a lot of ways. One way is an explicit mandate to find that a pre-determined conclusion is supported, even if the data don't support it. Another way is to place a greater emphasis on speed of delivery than on accuracy, and to avoid quantifying the true trade-off between the two by invoking the magic buzzword "business concerns" or "bottom line" or whatever.

Yet another way is that data analytics platforms are built from the ground up with hard-wired priorty given to scaling out the ability to test multiple hypotheses without any attempt to correct the significance metrics for the multiplicity of testing (or, even subtler, for subject researcher degrees of freedom that further affect the multiplicity of testing). Often, the business stakeholders who are demanding such an "analytics" system aren't even aware of the statistical fallacies they are inexorably baking right into the platform itself (one might call this the "Hadoop disease", though it's not stricly the fault of Hadoop or Hadoop-like tools).

At any rate, I would say in the current climate of "analytics" in business environments, to a good first approximation, one can assume that "make it easy to understand" is exactly equivalent to "throw out any and all difficult yet rigorous science until the thing is cheap and easy, and then just use that."

Thanks for your perspective. It is disheartening to see what you're talking about happen. I've definitely witnessed this first-hand and it was frustrating to me to essentially see lots of rigorous scientific work be dismissed because of someone's ego or because someone making the decisions couldn't wrap their brain around it.

What I've seen around this is analytics professionals hired under the pretense that their skills to produce accurate scientific conclusions will be used for the good of a business, yet having their conclusions and efforts dismissed for no good reason other than decision makers 'didn't get it' or otherwise just refused to heed the results. So why did they hire experts in the first place, then? To lend the company credibility that it doesn't really deserve? I'm sure lots of the reason for this type of thing is politically motivated, as you previously mentioned.