Asking asking an entity that can only discern its environment through continuous functions to explain itself is like asking an amoeba to play piano.

Machine Learning is a statistics renaissance, it is changing the world, but "Explainable AI" is a notion that is totally divorced from where the field is at atm.

Explaining ML inference is an active area of research, and there are plenty of decent approaches which are getting good results.

For example, LIME[1] can explain why a neural network doing visual classification came out with the result that it did. Distilling the Knowledge in a Neural Network[2] is another approach which leads towards simpler representations.

LIME isn't perfect, but that's the nature of research (and why DARPA funds this kind of thing).

this is a fool's errand

"Distilling the Knowledge in a Neural Network", authors: Geoffrey Hinton, Oriol Vinyals, Jeff Dean

Fools, all of them. DARPA should ask HN next time it wants to know about the current state of AI.

[1] https://github.com/marcotcr/lime

[2] https://arxiv.org/abs/1503.02531

You get some weird outputs from these sometimes though which makes it hard to automatically show them to users. For example, a reason might be "because you liked salad restaurant X you should check out check BBQ place Y" and it's because there happens to be an overlap in the users who like both captured in your model. Yet it can cause cognitive dissonance for the users who are either strictly healthy eaters (salads only, no BBQ) or delude themselves into thinking they are (forgetting how much they actually order BBQ in addition to salads). That's the main challenge I see -- figuring out how to filter out reasons from these approaches that don't jive with common intuitions or, even harder, get people to learn to trust the reasons as counter-intuitive as they may seem.

Tree-based classifiers always have the reputation of being "explainable". As you note this isn't always as simple as it should be.

In the unsupervised space I really like plotting dendrograms on hierarchical clustering. There's an excellent example in the recent DeepMoji paper[1] where they show how similar emojis cluster together AND how you can truncate the hierarchy at different depths to get capture different ranges of emotion.

It's laughable when people insist that DARPA are foolish for funding work in this area.

[1] https://arxiv.org/pdf/1708.00524.pdf

The goal is not necessarily to explain _exactly_ what's happening inside the AI, no more than a human can explain their decisions by describing _exactly_ what's happening inside their heads.

The explanation is an additional _decision_ that accompanies the original classification decision. The explanation space has its own utility function and the goal is to find the explanation with the highest utility.

But the notion of "Explainable AI" just reeks of upper middle management folks performing what they see as visionary risk management. When in reality these people don't know what they are talking about. Explaining exactly what's happening inside "the AI" is the domain of linear algebra. We can't be as exact in plain english but the generalization "it's fitting the training data" speaks volumes more than any _just_around_the_corner_ "explainable AI" secret sauce algorithm ever will.

AI says: "kill this person".

If it's only "upper management folks" asking "why" then I'm all about that visionary risk management!

Hell, I debug machine learning models every day, and most of my time is spent around working out "why". And the answer "it's just fitting the training data" is often wrong, insufficient, or misleading.

For example, I'm currently working on a sentiment-related task, and there's a particular class of misclassification where the model isn't taking the context of a word into account sufficiently.

In this case, the data is fit nicely, but the features it is fitting to don't contain the context. By adding tri-grams I can capture the context sufficiently to avoid this problem.

However I submit a notable exception! The vision system of self driving cars is a neural network that is currently changing the world and there is no lack of explication, we know exactly how it works. These "convolutional neural network" infer object boundaries from differences in intensity among matrices of pixels by scanning different patches of the pixel matrix and cross comparing these patches.

But you are right there is a lot of hype. I think part of the issue is the collective idolization of "algorithms". The limiting factor is data. And we don't have the data required to development models that are agent like in structure. Self interested agent systems, that employ subject-object language, have only been developed once. And they needed a training dataset measured in millions of years of evolution.

You say the limiting factor is data. But there has been a trend in the last 2 years to run ML on data generated from simulations (games, auto, chemical bonds, robots in VR, AlphaGo, etc). Simulations are dynamic datasets with unlimited flexibility. The better we learn to simulate, the closer we will get to reasoning. Both simulations and reasoning are based on object-relation graphs for describing the scene.

My vision is that we will build better, more precise simulators that would allow an AI to input a problem and run experiments and search ("what happens if"). Just like AlphaGo, but for the real world. A combination of neural nets for "intuition" and simulators for precision would solve the problem.

Simulators could be of many kinds: chemical molecule simulator, cell simulator, physical simulator, city traffic simulator, car simulator, flight simulator, and so on. Basically what's been the object of activity of supercomputing, but this time with neural nets selecting the experiments.

Neural nets are more like reflexes than reasoning. Most of them are feed forward and some don't even have memory and can't solve references. So it's unfair to expect a job that is best done based on graphs or on memory-attention to be done by a rudimentary system that only knows to map X to y.

But they are not totally unexplainable - you can get gradients on the data and see what parts of the input data most influenced the output, then you can perturb the inputs to see how the output would change.

I'm thinking... using chemical signals of some sort to force amoeboid cells to move in certain directions or take certain shapes, we can control gradiants of amoeba in sequence, and thus "teach" them encoded piano music.

But I'm not sure how this applies to AI.

https://www.youtube.com/watch?v=wfYbgdo8e-8

Explaining how an AI system makes decisions is completely unrelated to any kind of progress (or not!) towards AGI.

In fact, the department of defense is far behind even the department of energy inside the US government on high performance computing, algorithms and machine learning.

Second, the Department of Energy isn't just a random department with a lot of spare computing power. They run the National Labs, and part of that work is to make supercomputing available to academic and industrial researchers too. These are also used for nuclear-related projects (part of the DoE's mandate, but obviously related to DoD concerns too) and I would bet that there is some mechanism to boot off academic researchers if the national interest suddenly required exaflops of computation,

[0] https://www.fbo.gov/

I seem to recall that the DoE, NOAA, and NASA always had a lot of computing resources, and that they generally rented time on those clusters out to other government agencies.

I do wish the government would step up on AI though -- I think that they owe it to the country to put together a Manhattan Project for AI.

The Chinese government is leading (and funding) massive efforts in high performance computing and AI. If they win, entire high tech industries could be situated in (and lead by) China.

It actually works out better if the US government computers are through various science organization labs, because it locates much of their computing power in places also accessible to universities and at agencies amenable to US universities using their resources. So funding computers through the DoE, NOAA, NASA, etc actually makes them more available for the US research community, while allowing the military to lease the time they need (eg, for calculating explosion dynamics).

The total funding for US supercomputers also could be higher, but that's a general problem with US infrastructure investment.

My main point was there's no real concern with the computers being owned by scientific institutions instead of the military -- they're still government funded, it just makes them more widely accessible for use.

I hope we can come up with a better primary goal for AI than mass murder

Instead, they contract with academic and industrial researchers to work on specific things that are (or might someday be) of particular interest to the DoD. Thus, if there's a huge DARPA-funded breakthrough in Probabilistic Programming, it's going to have (say) Frank Woods' and Oxford University's name on it, not the DARPA program manager's. The final work might not even be published. They're happy to fund industrial groups if that lays the groundwork for products that the DoD eventually wants to buy.

https://cs.stackexchange.com/questions/37618/who-coined-the-...