I think there will be regulation that requires some users of AI to provide an explanation upon request. For instance, banks could be required to "explain" why you didn't get that loan. What if the decision is based on a credit score that includes some AI prediction that ultimately relies on the entire training corpus?
The bank can give you a list of factors that play into the decision but they may not be able to explain deterministically why a very similar customer did get that loan. At that point I think we're going to resort to statistics that prove a lack of bias against certain protected characteristics, but that's not really an explanation, is it?
I think we will never get useful and complete explanations for everything that AI does. Society will just accept some explanation-like thing or proxy and move on.
If they understood it 100%, what clarification is needed?
Your dog will never understand calculus or why Fourier transforms are interesting. There's almost certainly topics that are beyond human comprehension that an advanced artificial or alien intelligence can easily handle.