Essentially, the AI was showing how it came to a conclusion by, for example, highlighting parts of the image based on how much they contributed to a conclusion (and showed that a husky was mistaken for a wolf because of surrounding snow) or words which lead to context (showing that an AI to identify atheist vs theist texts was using email headers instead of the body).

This doesn't of course explain the entire process but it means a doctor for example can see that the AI concluded it's not the flu because while symptoms A an B were present, C wasn't there was a symptom X that is in the "Not flu" category and the doctor can then easily apply human intuition to say it's the flu after all because the patient is according to the medical record prone to X anyway.