If explanation, reasoning and thought processes are required then make them part of the job description. A detective who can't explain his action is unfit for the job. A software which can't explain it's recomendations is unfit for the job.

Btw, explanations given to judges have nothing to do with what really happend. A detective can tell a lie and the software will pick the explanation which is most likely to be acepted by the judges.

>and often the algorithms behind it are secret

and the algorithms used in brains are open source?