I’m no expert, but tools such as SHAP and DeepLift can give you insight into what activates a network. It’s probably not possible to inspect a network with billions of parameters, however it’s to be expected since I don’t think that explainable ML is an established field yet.

But also think about it from another angle: it doesn’t seem too hard to explain why people say what they say. We can usually get into the shoes if the other person if we try hard enough. However, if we say there’s no way for us to explain GPT-3, it just shows how fundamentally different it is from human mind.

Agreed. Even if we put research into deconstructing and attempting to understand how deep neural networks work in tasks such as autonomous driving, the fact is that these tasks are too complex to even logically describe.

That said, I do think it is possible to come up with robust guarantees to these methods.

Really? I bet in a few years we'll have tools that can inspect a model and tell you exactly what parts do what function and how they do it.