Given that eyewitness testimony is consistently unreliable, biased, and internally inconsistent, no, they shouldn’t. Human judgement is the best we can do so far, but it is hardly a worthwhile goal, especially with systems that take us from affecting individuals on a person-to-person level to influencing the interactions of millions of people with their government.

Algorithms can be explainable, and I agree that anything that affects your standing in the eyes of government should have that as a minimum requirement.