I think part of the problem here is what 'explainable' means and to whom. A Deep learning or Data Analyst may be looking for or accept one kind of answer (or even be able to comprehend it), while a Medical Doctor is looking for something else to hang his hat on, and the layman has an even different (lower?) expectation of explanaition.

Now perhaps we could design another set of algorithms (NN generated?) that would present the right type of explanation to each. Frankly, I'm not very interested in the layman's explanation, because what they want to hear is so disconnected from what's happening that it often seems irrelevant. That said, the placebo effect is real so you want the doctor and patient convinced that they are doing the right thing... so maybe it's more important (edit to add) for actual medical outcomes?