Because it limits your model to doing things you could understand, which thus makes it less powerful. In other words it’s a useful property only so far as you get it for free. A self driving car AI being understandable isn’t worth it killing more people in the field.
In some sense, it's possible to interpret any model, the effort requires just varies.
On your self driving car example, one which you can explain it's decisions to regulators is much more likely to be approved than one which you can't.
Given the amount of models I've built, I'm not sure there is a scarier thing than an uninterpretable model moving a ton of metal around at high speed.
It’s a vastly larger solution space. So it’s really the reverse that would be surprising.
> In some sense, it's possible to interpret any model, the effort requires just varies.
Models can be of arbitrarily large sizes to the point where people really can’t understand them. How do you go about dissecting a 2 layer NN with 10^40th nodes?
I'd probably take the output of the first layer, and cluster it.
It would very much depend on what this NN was attempting to do.
Like, all of the work in this field does suggest that people value this feature, and its incredibly useful for debugging which is generally pretty hard in ML/statistics.
> It’s a vastly larger solution space. So it’s really the reverse that would be surprising.
Imagine a world in which linear models returned the entire matrix by observation rather than the coefficients. People would argue that it was uninterpretable, but it's a problem of tools.
I actually think that if you can instrument a model appropriately, then you can definitely build an interpretation layer on top of it. Clearly that doesn't make the model perform worse.
Even if you can't instrument it, you can run thousands of experiments changing one feature at a time and then estimate the impact of this feature on the model. Granted, that's not practical on many problems, but neither were deep NN's a decade ago.
Any explanation method will give you more insight into how your model, features and data interact. This will allow you to improve the model, and also avoid insane features that are just capturing either information from the future or are not worth the processing effort.
Maybe it's that I came to data science from a theory-driven science point, but I don't really understand why you wouldn't want to be able to interpret your model.