But when an AI gets sufficiently complex of course there won’t be explanations that make sense for those kinds of errors, because just like a human the AI is integrating lots of different bits of information that it has learned are important and it has limited capacity and sometimes it just gets its attention focused on the wrong thing and it just didn’t see the guy, okay?
Demanding that AI be explainable is fundamentally demanding that it not be intelligent.
That out of the way, there are AI approaches that can explain their actions just fine without going dumb. For example, I posted this comment earlier:
https://news.ycombinator.com/item?id=30872400
about an AI system called Nook that recently won a tournament against 8 human champions of the card game Bridge. In Bridge, players must be able to explain their moves, so an AI player without the ability to explain its decisions can't play a full game of Bridge.
That is not the kind of explanation that is needed in AI systems. When people talk about "explainable AI", they literally just mean systems that can answer the kind of question that a human would be able to answer.
That's because a question that a human cannot answer is very likely to have an answer that a human will either not be able to understand, or will have to work very hard to understand... which is no better than no explanation