Stop Explaining Black Box Models for High Stakes Decisions

Re:Stop Explaining Black Box Models for High Stakes Decisions by TeMPOraL on Friday November 01, 2019 @07:44AM (#21417769)

Personally, I currently believe that this research is quite probably bunk. There was a recent HN thread in which it was pointed out that some of the results of experiments involved might have been data processing errors. And then there's the most obvious confounding factor - peoples' reported reasoning may not be their true, or complete, reasoning, because people aren't always comfortable with revealing all the details.

But regardless of that, while post-hoc explanations may not be that good for building predictive models of behavior, observed behavior is. We generally know how people behave, and we know the expected variance due to individual and situational circumstances. Truly unexpected behavior is rare in society, and we tend to filter it out. Truly unpredictable people get locked up and/or are given medical help, but even less unpredictable people are tested and kept out of professions and activities where that unpredictability could cause problems.

[ Parent | on HN ]

Re:Stop Explaining Black Box Models for High Stakes Decisions by carlmr on Friday November 01, 2019 @07:58AM (#21417836)

True, humans are more predictable and we can kind of probe at it until we reveal the truth. I guess we could do sensitivity analysis on ML models but most people aren't equipped for that.

>Truly unpredictable people get locked up and/or are given medical help, but even less unpredictable people are tested and kept out of professions and activities where that unpredictability could cause problems.

Given our knowledge about e.g. Narcissists or psychopaths their behavior is not totally unpredictable. I guess a serial killer could be extremely predictable, but not something we want in society.

[ Parent | on HN ]

Re:Stop Explaining Black Box Models for High Stakes Decisions by TeMPOraL on Friday November 01, 2019 @08:07AM (#21417878)

Variance reduction isn't the direct reason we lock away people or restrict their opportunities; personal and public safety is. So a serial killer is hunted and imprisoned, and work is done to detect would-be killers before they kill. Variance, however, is a safety issue in many areas - e.g. gun ownership, police service, flying, driving. So we have psychological evaluations not just to weed out the individuals obviously incapable of handling a task safely, but also the highly unpredictable ones.

[ Parent | on HN ]