Yeah, there's a good book on this very issue called "Weapons of Math Destruction."[1] The author did an interview on a podcast and discusses how algorithms used by these judges reflect our racial biases.[2] One example that I remember is that the algorithm takes into consideration one's zip code, which correlates strongly with race in the US.

[1]: https://weaponsofmathdestructionbook.com/ [2]: http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.ht...

You are right to be worried.

"It found that widely used software that assessed the risk of recidivism in criminals was twice as likely to mistakenly flag black defendants as being at a higher risk of committing future crimes. It was also twice as likely to incorrectly flag white defendants as low risk."

Source: https://www.nytimes.com/2016/06/26/opinion/sunday/artificial...

I agree wholeheartedly with this concern, but I think there is also a counter-argument too. If you assume we're starting from a situation where there is bias, then you probably want to look for ways to reduce that bias. The software may have an advantage here too.

It may be easier to objectively measure and correct for undesired (racial, or otherwise) bias in an automated system than it is to objectively measure and correct for bias in human judgement. And even if it's more difficult to correct for undesired bias in an automated system, at least that correction is more likely to be permanent and applied uniformly, compared to things like giving training courses for humans, which will vary in effectiveness across the recipients of the course, and will need to be refreshed frequently.

There are a lot of qualifications and "maybe"s in that argument though.

> ...other correlated factors will allow the algorithm to continue to enforce this racism, but now with legal immunity.

I'm sympathetic, but how do we keep from causing more harm than good here? It seems like blindly ignoring all data that correlates with race is also bad. We do this in some circumstances already (i.e., IQ tests when doing hiring or admissions) and I think it's a defensible point that we're worse off for it.

> blindly ignoring all data that correlates with race is also bad

This is a very theoretical, futuristic red herring. Race factors into decision-making far, far too much. It's like worrying that if someone developed a perfect, bug-free crypto library, then everyone would use it and we'd have a monoculture - not really a present concern, and in fact a good problem to have (nor is it an eloquent analogy, but I have to run).

I can't really tell if you are for or against algorithmic sentencing from this post.

I think there are fair existing examples both for and against it, so I'm not sure calling either position futuristic or theoretical is fair.

There was a Washington Post article and arXiv paper on this topic several months ago. [1][2] The problem boils down to the definition of bias. Northpointe, the company that owns COMPAS, says the algorithm treats black and white defendants equally, which is true. Others claim that its predictions for blacks are wrong more often than for whites, which is also true. The main point of the paper is that you can't have your cake and eat it too. If you don't want to consider race, then the predictions will be wrong more often for blacks. If you don't want the predictions to be wrong more often for blacks, the algorithm has to consider race. (I'd recommend the articles for a better explanation.)

In my mind, the ownership of the software is what's troubling. First, it's not public, so nobody can look at the guts. And second, the Loomis decision constructs even more walls around the software. It's hard for people to critically examine and change something they can't see.

[1] https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/1... [2] https://arxiv.org/pdf/1609.05807.pdf

Chris Stucchio wrote an interesting blog post which changed my mind on this. Instead of exacerbating it, a Machine Learning model can actively correct for bias.

https://www.chrisstucchio.com/blog/2016/alien_intelligences_...

There are some issues with the arguments contained in that post, which I discussed here: https://news.ycombinator.com/item?id=12630980

Bias correction is possible, but, tautologically, a model whose structure cannot capture a bias cannot correct for that bias. This means that a modeller must either understand the bias, and accommodate for it in their model, or use a model that might be able to capture unknown structures and run the (serious) risk of over-fitting that model.

So, when we train all the AI and robots that are going to take over all of our jobs in the next 5 years, we need to make sure we are very careful.

" ... other correlated factors will allow the algorithm to continue to enforce this racism"

Which correlated factors are you thinking of?

Some proxies for race: - Where somebody lives - Their name - What sports teams somebody supports - Their finances - consumer preferences - education level - what medical conditions somebody has - political leanings

I'm not saying these factors can't legitimately be used in a risk assessment, but they could be used to make a good bet on race.

Finances? While it is true that blacks and native americans have the highest poverty rates, if you take a random poor person they are about 50% more likely to be white than black. Same with the many of your other "proxies". Therefore they would be a poor bet for race.

If you were using these proxies to identify the race of a person jailed for a crime, they might be good predictors, but only marginally more useful than just blindly guessing black, since they constitute the majority of incarcerated people.

I think, by attempting to remove the smoke screens a "real racist" would use, you give them a new one - "look, our opponents want to ignore actual data".

I don't see how those factors are relevant at all, regardless of the race of the criminal. The problem to address is the lack of transparency. Trying to guess what inputs are being factored in is pointless.

Even if you are careful to remove all features you think could tell the race (let's say geographic location etc), it could still be giving harsher sentences to minorities if those are treated differently upfront. For instance, it would make sense for the model to learn that the more charges count the harsher the sentence. This sounds right: a bank robbery should be less punished than a bank robbery + a carjacking, at least in the US judicial system.

But now let's say minorities gets a "resiting arrest" charge on top of their original charge more often, because of many factors such as the police bias, the bias of the minorities towards police, etc. (By bias here I mean all spectrum : racism, but also fear of the police etc).

As long as minorities are treated differently upfront, then the model would treat them differently.

But if all of that is true, the model doesn't change the equation. The problems you outline need to be addressed upstream. Trying to account for and correct them at the point of sentencing is very problematic.

On the other hand you could probably find ways to make the algorithms less racist than their human equivalent. No system is totally unbiased but doing better than what we have is possible.

Not to mention that discrimination in hiring decreases opportunities for minorities. Which in turn increases recidivism.