A detective can explain their reasoning and thought processes, and a judge / jury / review board / whatever can make a determination based on that whether or not the detective acted properly.

Software doesn't defend itself, and often the algorithms behind it are secret. Even when they aren't, many of the models created today are just not explainable, even by those who have developed and trained them.

If explanation, reasoning and thought processes are required then make them part of the job description. A detective who can't explain his action is unfit for the job. A software which can't explain it's recomendations is unfit for the job.

Btw, explanations given to judges have nothing to do with what really happend. A detective can tell a lie and the software will pick the explanation which is most likely to be acepted by the judges.

>and often the algorithms behind it are secret

and the algorithms used in brains are open source?

It's not perfect, but at least a detective can be deposed.