I worked on making a more explainable AI in bio for while. Probably a lot of the unexplainable AI that is being discusses is about deep learning. Some of the biggest advances in deep learning can also be said to have made the models more explainable. Take computer vision, for example. We have great success with convolutional neural networks because these move the models away from unexplainable fully connected layers where the mathematicals naturally produces many neurons with small effect sizes and towards detecting larger coherent structures in the convolutional layers.

I agree there is more work to be done and making good mathematical decisions when designing the layers seems like a promising way to go.