Is that because you care about verifiability though, or actually interpretability? If the system provides you a diagnosis without context, but later can be confirmed by actually testing for that condition, do you still refuse treatment because you don't know how the original diagnosis was suggested?

Oh, if it's a recommendation for running some tests, I can't imagine much pushback unless they're particularly invasive or expensive. It's not like doctors don't order large panels of tests at the drop of a hat.

If you then have treatment based on those tests, that's really a doctor being assisted by an AI at that point.

The current explanation for anything in medicine is "Dunno, we tested it and it seems to work". It is a much better one than "The computer says so", but it's one the computer can tell you too.