Yep. "Explainable AI" is an active area of research with huge amounts of funding and interest from institutions in the US, EU and China. For example, this is the DARPA programme:

https://www.darpa.mil/program/explainable-artificial-intelli...