> If you know how the algorithm works (and this one must be open source)If you know how the algorithm works (and this one must be open source)

Hah, no chance any of the sources to these things will ever be public.

Well, just throw the constitution in the bin then (any democratic constitution, really). Might be fine for dictators though, getting rid of those unreliable judges.

It has been shown many times that if you know the human making the decisions, you can game it by knowing the person, or anything from having a preexisting relationship with them, being famous, dependence (e.g. they're a tenant of you, or a landlord), planting ideas, ... Most of these are not illegal, or even possible to avoid.

And then, of course, there's racism, nepotism and outright bribery.

Humans, under the best of circumstances, are known to be unfair. Can an algorithm really be that much worse ?

You miss the point. We don't have a computer process that establishes factors such as the level of remorse expressed by the offender or their claimed motive, whether their misinterpretation of the relevant law is plausible or not, or the magnitude of the harm caused to the victim or whether the plaintiff's claim of monetary losses suffered is plausible or not so the "algorithm" is a thin veneer over a human determining which factors are and aren't relevant... so you get exactly the same level of bias before plus noise from the human judge making these determinations not knowing exactly how the algorithm works and a false claim of greater impartiality.

It's probably even worse if you try to pull the human out the loop altogether and throw an ML black box at the testimony, which doesn't understand the actual situation the defendant describes at all but does pick up the statistical association between African American vernacular words used and higher sentences in its training corpus. Much easier to train a model that picks up the prejudice in sentencing factors than the situational nuance...

I think the point is that we are currently using opaque, poorly trained and erratic ML (Meat Learning) algorithms. Oh sure the meatbox says that it's allocating N years for this, M years for that, but empirical studies have proven that this readout is nonsense.

Which kind of ML is easier to train to be non-racist? I don't know, but it's not obvious to me that meat is the winner here.

Exactly this. Washington Post had an article sometime ago about software used in sentencing/parole that consistently judged black defendants more harshly (recommended longer sentences than white people for the same exact crime and recommended parole at a lower rate for black prisoners). The software was proprietary and there was no access to the code but even if it were perfectly written it learned from the data fed to it.

The software came to the conclusion that race (black vs white) was a predictor of crime and recidivism. How did it come to this conclusion? Because of the data

- Black people get arrested at a higher rate than white people

- Black people were also more likely to re-offend than white people

So the conclusion that the program came to makes sense. But it totally ignores the external factors that lead to the 2 above statistics

- Black neighborhoods are more heavily and more aggressively policed, meaning that you will uncover more crimes.

- Black people are targeted by police (black and whites consume drugs at the same rate, but black people are more likely to get stopped, more likely to get searched and more likely to get arrested)

- Over their lifetime, black people are more likely to have more contact with police (even if they live in an all white neighborhood). All it takes is for one of those times they commit a minor infraction (having weed on them etc). That conviction then becomes a justification for elevating their 'risk' level.

- Black ex-convicts have a harder time getting jobs due to inherent bias (all ex-convicts have a hard time, but black ex-convicts have a much harder time). This closes off avenues to gainful employment making turning back to crime one of the few options available to them. And once again, black ex-convicts have more contact with police than white ex-convicts.

Sans context, the input data can be an incredibly effective way of propogating bias into the model.

Even strong AI will not solve for this without some corrections/mintigating strategies. The most effective strategy would be solving the policing problem (bias, over-policing, how we close off all avenues at rehabilitation to people who have been convicted of victimless crimes)