Re:Interpretability in Machine Learning: An Overview
by brazzy on Monday November 30, 2020 @06:34AM
(#25252759)
> So if learned models give us access to some truths without access to their (human intelligible) explanations, I think we need to just embrace that.
The question is: if the model is not interpretable or understandable, how do you know that what it gives you is, in fact, truth?
You basically need some kind of external validation of the results. In the case of Go, the rules of the game and competion basically provide that in a very authoritative way. I don't think this is the case for all that many domains.
As I see it, behind the desire for interpretability there are two main concerns:
* could a model that gives really good answers in all the cases we have tested still give catastrophically wrong answers in some cases we have not foreseen?
* could the model be relying on some flaw or bias in the training data which we haven't realized?
The question is: if the model is not interpretable or understandable, how do you know that what it gives you is, in fact, truth?
You basically need some kind of external validation of the results. In the case of Go, the rules of the game and competion basically provide that in a very authoritative way. I don't think this is the case for all that many domains.
As I see it, behind the desire for interpretability there are two main concerns:
* could a model that gives really good answers in all the cases we have tested still give catastrophically wrong answers in some cases we have not foreseen?
* could the model be relying on some flaw or bias in the training data which we haven't realized?