Also, science is unfortunately full of magic numbers - but it is pretty amazing when you feed in pendulum motion data and the resulting equation ends up being Newton's second law: http://phys.org/news/2009-12-eureqa-robot-scientist-video.ht...
Do you think people should stop working on it?
Most interesting things seem difficult to measure, until someone finds a way, and then it seems obvious. An example is search engine quality. At first this might seem too subjective to be measurable. But Google started measuring search quality using a panel of humans, and now everybody does that.
The whole idea of these challenges is to broaden the search, to hope for key insights that by their nature seem elusive at first.
The problem with unexplainable AI isn't just the lack of explanations. It's the fact that the entire model is a black box, so you cannot somehow tweak a single part of it without retraining.
But still, the results are very difficult to interpret. Yes you can verify that the equation works, that it predicts the data. But why does it work? Well who knows? No one can answer that. Understanding even simple math expressions can be quite difficult. Imagine trying to learn physics from just reading the math equations involved and nothing else.
One biologist put his data into the program, and found, to his surprise, that it found a simple expression that almost perfectly explained one of the variables he was interested in. But he couldn't publish his result, because he couldn't understand it himself. You can't just publish a random equation with no explanation. What use is that?
I think the best method of understanding our models, is not going to come from making simpler models that we can compute by hand. Instead I think we should take advantage of our own neural networks. Try to train humans to predict what inputs, particularly in images, will activate a node in a neural network. We will learn that function ourselves, and then it's purpose will make sense to us. Just looking at the gradients of the input conveys a huge amount of information of which inout features are the most and least important. And by about how much.
But mostly I think the effort towards explainability is fundamentally misguided. In the domains where they are supposedly the most desirable, like medicine, accuracy should matter above all. A less accurate model could cost lives. Accuracy is easy to verify through cross validation, but explainability is a mysterious unmeasurable goal.