Stop Explaining Black Box Models for High Stakes Decisions

Re:Stop Explaining Black Box Models for High Stakes Decisions by TeMPOraL on Friday November 01, 2019 @08:23AM (#21417984)

I'm talking about variance. We roughly know what to expect of other people, we have a good idea of the shapes of distributions of human behaviour. We know the range of expected behaviours in a given situation with high confidence. We know how stupidity looks like.

None of this applies to ML. ML models can fail in ways we can't easily predict, for reasons we don't expect. Viewed as minds, they run on a different architecture and on different firmware than human minds. They're alien to us. Alien like a kitten who suddenly freaks out for no reason, like ants that can get stuck walking in a loop - except more so, because we've know cats and ants for as long as humanity exists, and they're more similar to us than ML models hardware- and firmware-wise.

[ Parent | on HN ]

Re:Stop Explaining Black Box Models for High Stakes Decisions by jhrmnn on Friday November 01, 2019 @09:13AM (#21418332)

Ok, fair point. So I guess whereas one could argue the explainability problem away, the predictability of failure modes is non-negotiable.

[ Parent | on HN ]

Re:Stop Explaining Black Box Models for High Stakes Decisions by TeMPOraL on Friday November 01, 2019 @09:42AM (#21418641)

Perhaps, but explainability is very helpful for predictability - it allows us to do better than just making statistical inferences from observed behaviour. Since ML models, unlike biology, are entirely our design, we have an opportunity to make them explainable and reap the benefits in terms of predictability.

[ Parent | on HN ]

Re:Stop Explaining Black Box Models for High Stakes Decisions by mattmcknight on Friday November 01, 2019 @09:28AM (#21418508)

You would be mistaken to trust a random expert without some independent verification of the accuracy of their predictions.

[ Parent | on HN ]