One of the things neuroscience and psychology showed us is that humans always have a post hoc reasoning they can explain, but it's not always the actual reason they did something.

So humans explaining themselves is often misleading data, which may be worse than no data.

Personally, I currently believe that this research is quite probably bunk. There was a recent HN thread in which it was pointed out that some of the results of experiments involved might have been data processing errors. And then there's the most obvious confounding factor - peoples' reported reasoning may not be their true, or complete, reasoning, because people aren't always comfortable with revealing all the details.

But regardless of that, while post-hoc explanations may not be that good for building predictive models of behavior, observed behavior is. We generally know how people behave, and we know the expected variance due to individual and situational circumstances. Truly unexpected behavior is rare in society, and we tend to filter it out. Truly unpredictable people get locked up and/or are given medical help, but even less unpredictable people are tested and kept out of professions and activities where that unpredictability could cause problems.

True, humans are more predictable and we can kind of probe at it until we reveal the truth. I guess we could do sensitivity analysis on ML models but most people aren't equipped for that.

>Truly unpredictable people get locked up and/or are given medical help, but even less unpredictable people are tested and kept out of professions and activities where that unpredictability could cause problems.

Given our knowledge about e.g. Narcissists or psychopaths their behavior is not totally unpredictable. I guess a serial killer could be extremely predictable, but not something we want in society.

Variance reduction isn't the direct reason we lock away people or restrict their opportunities; personal and public safety is. So a serial killer is hunted and imprisoned, and work is done to detect would-be killers before they kill. Variance, however, is a safety issue in many areas - e.g. gun ownership, police service, flying, driving. So we have psychological evaluations not just to weed out the individuals obviously incapable of handling a task safely, but also the highly unpredictable ones.

"humans explaining themselves" is entirely different thing than "humans understanding human behaviour". You may have rationalized your choices, but I understand both your choices AND that you rationalized them.

But do all those techniques actually work better for humans than trying to post-hoc explain machine learning? Perhaps both cases are just about inventing stories around the decisions so they can be perceived as relatable.

I'm talking about variance. We roughly know what to expect of other people, we have a good idea of the shapes of distributions of human behaviour. We know the range of expected behaviours in a given situation with high confidence. We know how stupidity looks like.

None of this applies to ML. ML models can fail in ways we can't easily predict, for reasons we don't expect. Viewed as minds, they run on a different architecture and on different firmware than human minds. They're alien to us. Alien like a kitten who suddenly freaks out for no reason, like ants that can get stuck walking in a loop - except more so, because we've know cats and ants for as long as humanity exists, and they're more similar to us than ML models hardware- and firmware-wise.

Ok, fair point. So I guess whereas one could argue the explainability problem away, the predictability of failure modes is non-negotiable.

Perhaps, but explainability is very helpful for predictability - it allows us to do better than just making statistical inferences from observed behaviour. Since ML models, unlike biology, are entirely our design, we have an opportunity to make them explainable and reap the benefits in terms of predictability.

You would be mistaken to trust a random expert without some independent verification of the accuracy of their predictions.