No, you can challenge the validity of the risk assessment. Bayes is used in jury trials all the time.

https://en.m.wikipedia.org/wiki/Bayesian_inference#Applicati...

What we have today is institutional racism based on people’s gut reactions. An NN can be tested and shown that given all other factors it risks race, or any other protected class higher than it should.

> What we have today is institutional racism based on people’s gut reactions.

Institutional racism is not based on people's gut reactions. Institutional racism is based on flawed algorithms. Not computer algorithms in this case, but bureaucratic ones.

I don't see how a flawed computer algorithm would be any easier to correct than a flawed bureaucratic algorithm.

Because we can black box test the “risk assessment” that whoever sells to police departments. We can’t black box test bureaucracies.

Testing the results of the algorithms doesn't address the problem of institutional racism, though. Any bureaucracy will simply use the results that the algorithm supplies in a way that maintains the behavior that the bureaucracy is already programmed to maintain.

To address things like institutional racism, you have to change the institution. That's a human problem for which there is no technological solution (although technology can certainly be one of the tools used to create a larger change).

We can black box test bureaucracies just as poorly as we can software. For both, all we can do is measure inputs, and outcomes.

Since we've not made much traction on blackbox testing bureaucracies, I'm going to go ahead and say that blackbox testing algorithms will be even worse. (For some dumb reason, people are inclined to trust algorithms.)