The pneumonia project referenced in the article had Caruana voting against implementing the most accurate model: A neural net. Instead they went with a way less accurate logistic regression model, one they could safely implement in production, inspect, explain, and defend to the doctors.
Nobody does quite know why neural networks work so well. There is a Nobel Prize waiting there for someone or some team to solve this with mathematical (or physics) rigor.
Nodes in neural network layers can represent multiple features, or share feature representations. Do we know if a neural net (and which part) is targeting skin color, or acne? Do we know that credit risk models are targeting sex, even though we left out this feature (it may infer this from other features)?. Depending on the application, this is important to know for certain.
Reverse engineering sure does work, but can we fully find out the source code from a program, just by fuzzing inputs and looking at outputs? Or are we only looking at (perhaps a small part of) its behavior?
> You will run into trouble if you don't know what you're doing
Likewise: You will run into trouble, if you don't know for sure what your models are doing.
http://blogs.wsj.com/digits/2015/07/01/google-mistakenly-tag...
> as per common sense
https://en.wikipedia.org/wiki/Commonsense_reasoning#Commonse...
What I meant by "Because ML systems are opaque, you cannot really reason about what they do" (perhaps I was not being clear) was this: You can indeed _observe_ what they do, but being able to actually inspect the source makes verification much easier and more reliable: You know what parts of the logic you have covered, you can think of "danger areas" (and direct testing to them), and you can simply check whether all the cases you can think about have been covered in the source.
With opaque systems, you have no idea whether e.g. your ML-based autonomous vehicle will recognize people-painted-on-a-bus as people-on-the-road, until you actually test for that. And then you have to test people-painted-on-a-truck.
What I meant regarding "modular verification" is that you can check sub-modules according to some spec (or at least according to informal comments). This is quite different from what you can do when analyzing NN layers.
I suspect you are right in claiming (in your last paragraph) that one will always have to choose between "more understandable" and "more accurate". But I think we can do various things (the DARPA suggestion being one of them) to make the "more accurate" solution more verifiable.
Nobody understands exactly how painkillers (aspirin, paracetamol etc.) work on the molecular level. Yet they are generally held to be quite useful.
Edit: electroconvulsive therapy is an even more extreme example; we have no clue how it works, but it's very effective on severe depression. We only know it works because some Italians back in the olden days, before ethics committees were invented, decided to electro-shock a bunch of "crazy people" to see what happened. The reason they tried it was that electroshocking of pigs for slaughter had been observed to give a temporary anaesthetic effect.
> As exciting as their performance gains have been, though, there’s a troubling fact about modern neural networks: Nobody knows quite how they work. And that means no one can predict when they might fail.
Nonsense! Cross validation. Develop hypotheses, develop subsets of data to prove or disprove given hypotheses, observe how the network reacts. All of these people complaining about not being able to understand what's going on are either reporters, bloggers, or machine learning dabblers looking to say something seemingly unconventional.
From your linked article, which gives more specifics as to the argument:
> Because ML systems are opaque, you cannot really reason about what they do.
Yes, it is possible to reason about a system even if it is "opaque"; the discipline is called reverse engineering. Or the scientific method.
> Also, you can’t do modular (as in module-by-module) verification.
You can do "modular verification" in a variety of ways. Start with analyzing the behavior of each layer and how that changes as you incorporate more layers. It's beyond the scope of this comment to go into it beyond surface level, but there are a lot of papers written about it, google "analyzing neural network hidden activations" or something.
> And you can never be sure what they’ll do about a situation never encountered before.
Humans can never be sure what they'll do about a situation they haven't encountered. Or engineers. They can simulate the events that they can think of, but we can also do that with a neural network.
> Finally, when fixing a bug (e.g. by adding the buggy situation + correct output to the learning set), you can never be sure (without a lot of testing) that the system has fixed “the full bug” and not just some manifestations of it.
Fixing the "full bug" is often not something that can be done in traditional software development "without a lot of testing". Machine learning works the same way.
If you want if/then statements, use a decision tree. If you want strong accuracy on predictions, use a neural network. It helps to know what you're doing when verifying results. You will run into trouble if you don't know what you're doing, as per common sense.