Because we can black box test the “risk assessment” that whoever sells to police departments. We can’t black box test bureaucracies.

Testing the results of the algorithms doesn't address the problem of institutional racism, though. Any bureaucracy will simply use the results that the algorithm supplies in a way that maintains the behavior that the bureaucracy is already programmed to maintain.

To address things like institutional racism, you have to change the institution. That's a human problem for which there is no technological solution (although technology can certainly be one of the tools used to create a larger change).

We can black box test bureaucracies just as poorly as we can software. For both, all we can do is measure inputs, and outcomes.

Since we've not made much traction on blackbox testing bureaucracies, I'm going to go ahead and say that blackbox testing algorithms will be even worse. (For some dumb reason, people are inclined to trust algorithms.)