I've done some experiments with LIME -- it's one of the most promising approaches for extracting prediction reasons from arbitrary and otherwise opaque models. Another interesting approach specific to random forests is decision paths: http://blog.datadive.net/interpreting-random-forests/

You get some weird outputs from these sometimes though which makes it hard to automatically show them to users. For example, a reason might be "because you liked salad restaurant X you should check out check BBQ place Y" and it's because there happens to be an overlap in the users who like both captured in your model. Yet it can cause cognitive dissonance for the users who are either strictly healthy eaters (salads only, no BBQ) or delude themselves into thinking they are (forgetting how much they actually order BBQ in addition to salads). That's the main challenge I see -- figuring out how to filter out reasons from these approaches that don't jive with common intuitions or, even harder, get people to learn to trust the reasons as counter-intuitive as they may seem.

Yes, there is plenty of work to do.

Tree-based classifiers always have the reputation of being "explainable". As you note this isn't always as simple as it should be.

In the unsupervised space I really like plotting dendrograms on hierarchical clustering. There's an excellent example in the recent DeepMoji paper[1] where they show how similar emojis cluster together AND how you can truncate the hierarchy at different depths to get capture different ranges of emotion.

It's laughable when people insist that DARPA are foolish for funding work in this area.

[1] https://arxiv.org/pdf/1708.00524.pdf