We have hundreds of thousands of years (if not more) of trying to understand other humans behind us. It's encoded in our biology (theory of mind, mirror neurons), and more recently, in countless of disciplines we engage in - from arts, through law, to psychology. Truly unexpected human behavior is a very rare thing.

We ought to be able to do better than building our ML models as black boxes and saying "humans are black boxes too". We're engineering this technology, so we have an opportunity to make it inspectable. We should take it.

One of the things neuroscience and psychology showed us is that humans always have a post hoc reasoning they can explain, but it's not always the actual reason they did something.

So humans explaining themselves is often misleading data, which may be worse than no data.

Personally, I currently believe that this research is quite probably bunk. There was a recent HN thread in which it was pointed out that some of the results of experiments involved might have been data processing errors. And then there's the most obvious confounding factor - peoples' reported reasoning may not be their true, or complete, reasoning, because people aren't always comfortable with revealing all the details.

But regardless of that, while post-hoc explanations may not be that good for building predictive models of behavior, observed behavior is. We generally know how people behave, and we know the expected variance due to individual and situational circumstances. Truly unexpected behavior is rare in society, and we tend to filter it out. Truly unpredictable people get locked up and/or are given medical help, but even less unpredictable people are tested and kept out of professions and activities where that unpredictability could cause problems.

True, humans are more predictable and we can kind of probe at it until we reveal the truth. I guess we could do sensitivity analysis on ML models but most people aren't equipped for that.

>Truly unpredictable people get locked up and/or are given medical help, but even less unpredictable people are tested and kept out of professions and activities where that unpredictability could cause problems.

Given our knowledge about e.g. Narcissists or psychopaths their behavior is not totally unpredictable. I guess a serial killer could be extremely predictable, but not something we want in society.

Variance reduction isn't the direct reason we lock away people or restrict their opportunities; personal and public safety is. So a serial killer is hunted and imprisoned, and work is done to detect would-be killers before they kill. Variance, however, is a safety issue in many areas - e.g. gun ownership, police service, flying, driving. So we have psychological evaluations not just to weed out the individuals obviously incapable of handling a task safely, but also the highly unpredictable ones.

"humans explaining themselves" is entirely different thing than "humans understanding human behaviour". You may have rationalized your choices, but I understand both your choices AND that you rationalized them.

But do all those techniques actually work better for humans than trying to post-hoc explain machine learning? Perhaps both cases are just about inventing stories around the decisions so they can be perceived as relatable.

I'm talking about variance. We roughly know what to expect of other people, we have a good idea of the shapes of distributions of human behaviour. We know the range of expected behaviours in a given situation with high confidence. We know how stupidity looks like.

None of this applies to ML. ML models can fail in ways we can't easily predict, for reasons we don't expect. Viewed as minds, they run on a different architecture and on different firmware than human minds. They're alien to us. Alien like a kitten who suddenly freaks out for no reason, like ants that can get stuck walking in a loop - except more so, because we've know cats and ants for as long as humanity exists, and they're more similar to us than ML models hardware- and firmware-wise.

Ok, fair point. So I guess whereas one could argue the explainability problem away, the predictability of failure modes is non-negotiable.

Perhaps, but explainability is very helpful for predictability - it allows us to do better than just making statistical inferences from observed behaviour. Since ML models, unlike biology, are entirely our design, we have an opportunity to make them explainable and reap the benefits in terms of predictability.

You would be mistaken to trust a random expert without some independent verification of the accuracy of their predictions.

humans can be held accountable for their decisions though

This point is really about incentives: since we know that our fellow humans are held accountable, we know they have the incentive to make high-quality decisions that will not impact the world negatively. Therefore, it's easier to trust them, even when they go with their gut.

We can't project the same incentive structure onto software companies. They are a different scale than humans, and might be able to tolerate the possible hit to their reputation better than a human can. Their incentives are usually money-based rather than social-based.

And for the models themselves, they have no incentives, unless you count "reduce their error rate" or "be interesting enough for researchers to continue to research them". We barely know why they work. Our basis for trust is rather tenuous.

In this case the analogy would be that software companies should be hold accountable for the decisions their AI makes?

The article makes the case that if an interoperable model (one we can explain) isn't used, then the user of the black box model should have the burden of proof to prove that no interoperable model exists that does the job, and some level of responsibility for trying to develop one.

Only because we all agree that a human is a ‘person’ ie. a thing that can take blame.

Fascinating. I'd never thought of defining a 'person' this way.

You might have never been in a VP+ position in a company then. A lot of tasks in these positions are about avoiding consequences or shifting blame for actions that are either necessary but look bad or that had to be made with way too little input information. And not necessarily just one's own actions.

I was definitely thinking about corporate blame-shifting when I wrote that, and actually nearly said "legal 'person'" instead.

So make these ai’s corporations — instant personhood.