asamiam

I think this falls under the general "don't threaten the relationship." Whether it's joking about firing, saying "maybe we should just get divorced", joking a child was an accident etc. When things get stressful people often reevaluate comments in their new emotional state and what might have felt like a joke could become a threat. Subsequently instead of working to solve a problem for a situation they trust, people often work for self preservation.

We want our models to be opaque in the same way we like to ask people why they made a certain decision or act in the way they do. What's interesting to me is if you want to know about a person you'll get better information by asking their closest friends than themselves.

Perhaps there's something in the black box nature of our own self understanding similar to these hyper complex function approximators.

Judges make harsher rulings when they're hungry. A regulator might decline a loan and justify it b/c they had a bad experience with someone that reminds them of the person they're dealing with. AI makes bad decisions in boundary conditions or when they're trained on biased data. I think it's good to strive for opacity in every case but in complex decision spaces I'm not sure if we'll ever get fully satisfactory explanations.

Which is a meandering way of making the non-point that models making judgments have made me re-evaluate what I trust and why, and I think in many cases I'd rather trust a black box if I know what's gone into it and what's come out.