We make complex reasoned plans like "how to make coffee": I'm going to need a filter in the pot, I'm going to need ground coffee, water, etc; let's make sure the filter's there before I add the coffee and both before the water, etc." Then we use some sort of hierarchical planning heuristics to run it. If someone asks what I did I can explain it at varying levels of resolution ("I made coffee", "I got these pieces together (xxx) and then made coffee...") again depending on various "explanation heuristics" which we learned as part of being eusocial organisms.
These plans are complex (even "get out of bed and pee" is a pretty complex plan).
However below that are a ton of decisions we aren't even aware we're making. I'm convinced (and my software reflects) that the actual "plans" we make organically are very short, and that the interesting plans -- the ones we can talk about and that we typically care about -- are very abstract ones. Making coffee is super abstract, after all.
Perhaps for an analogy: the extremely abstract reasoning for not using an antibiotic is at the level of "chemistry" while what I'm talking about is at the level of "physics".
Similarly, the output of an opaque and perhaps unreliable process could be not just an answer but also a justification. If the justification can be verified more easily than starting from scratch, we've made progress.
Even in the humanities there is a difference between argument from authority (blind trust) and argument based on reason and justification. And sometimes we have both. Court decisions include not just the decisions but also an explanation of why the court ruled that way. It certainly doesn't eliminate bad decisions, but it's useful.
So the discipline of being able to explain yourself seems pretty important. I don't think AI will be able to fully participate in high-stakes decision-making until it can do that.
Humans have a very good intuitive understanding of how other humans think, even when that thinking is irrational. This understanding has evolved over millions of years and it even predates our species. We developer all kids of mechanisms for controlling irrational behavior and tested them for thousands of years.
Taking a new system of machine decisions and adding it to the mix sounds problematic.
I think this relates to the current discussion. You don't need an explainable model for what cereal you choose, but you need a 100% clear one on what tenants you choose, who you lend to, who you hire, etc.
We don’t do this for complex plans where we track multiple intermediate goals, even not on a piece of paper, but the intermediate gaps between what is written down are truly inscrutable.
That being said I have some sympathy for the technical portions of her argument (and for the rest too).