> If the software is accurate at picking choices that match the set goals, then it would seem that: the software is doing its job and is 'blameless', the racism is on the side of those that set the goals and/or the people claiming 'racism' because a person of a given skin colour was selected.

Well, no.

It's not enough to simply decline to input racial data into the machine-learning system. The system may discover a proxy for race, and make decisions based on that proxy. Name and address, for instance, may be enough to give a good idea as to someone's race.

We expect human judges and parole-boards, for example, to discount racial information, even if there are demonstrable correlations associated with race. If the machine-learning system is not built with that discounting in mind, it isn't going to do so of its own accord, as it was of course built to detect and exploit correlations.

> If it could be shown that there's low success in the software, yet it's relied on despite racial bias, then I would think few rational people could argue for its removal.

I don't follow. If the software has been unsuccessful, that seems like a good reason to remove it. If, additionally, it has been making decisions on racial grounds, that seems like another good reason to remove it.

> Race seems like a red-herring until it's shown that the metrics support poor performance of these systems.

As I hope I've made clear above, this isn't right. It's not just about performance, it's also about ensuring the system does not make decisions on the basis of an individual's race.

>> If it could be shown that there's low success in the software, yet it's relied on despite racial bias, then I would think few rational people could argue for its removal.

> I don't follow. If the software has been unsuccessful, that seems like a good reason to remove it. If, additionally, it has been making decisions on racial grounds, that seems like another good reason to remove it.

I mistyped there. I meant to say "few rational people could argue AGAINST its removal".

I had originally edited my sentence partially and mangled the meaning entirely.

I'm confused here, I read the article and it was somewhat muddy about the point trying to be made other than "machine bad!".

Racially biased could mean two things: despite not taking into account race the algorithm groups results racially at a statistically significant rate, or the algorithm is outright using race as a factor.

I assumed the former because it was not made clear in the article what was being talked about. Half the article is about things that may/may not use racial information (it's not said), and half the article is about things that do.

Are we talking about the same thing here?

A good example of this that I saw recently was AI used for underwriting health insurance. In order to make the algorithm "unbiased" in terms of race they removed race from the input dimensions. It then started heavily biasing along "frequency of doctor visits" which turns out to have a strong relationship with race. My takeaway is if equality of outcome is part of the goal that has to be part of your training as opposed to attempting to ignore the input, which seems obvious in hindsight.

If you're looking for equality of outcome, why go to the trouble of training a system in the first place? A pre-selected distribution applied arbitrarily is just as good.

Frequency of doctor visits seems a good indicator for determining how likely an insurance company will have to pay on a policy. If you want equal outcome, you end up setting a higher premium for everyone, and skip the underwriting altogether. The next part of the discussion would be subsidizing the higher premium for those that can't afford it. Just keep in mind that because there are fingers on the scale, fewer people can afford the "equalized" insurance than could afford the honestly evaluated coverage. (Prices are higher for everyone, so the pool of subscribers ends up being smaller if subscribers can opt out.)

It becomes a never-ending cycle of controls and impositions leading to collapse. Have the realistic underwriting, lower prices, and subsidize those (overall) lower premiums when needed.

Why should the system have access to the candidates' name?

You've been denied parol because someone with a name similar to you skipped bail is unfair regardless of race.

You've been denied parol because you come from a rough neighborhood is also unfair, regardless of race.

In fact, even if 99.9% of all people named X skipped bail, it would be unjust to deny somone bail because his name is X.

I think the true question is how you decide if an algorithm is actually biased or not. At a first shot I would suggest something like this: Divide some otherwise randomly sampled testing data into two sets, depending on the characteristic in question. If the ROC curve is approximately the same for both sets, I'd guess you could say the algorithm is not biased.

By Goodhart's Law, there does not exist such a metric of success. All success metrics are arbitrary in this situation.

Software is written by people. Racist software comes from racist beliefs. A belief that a neural network will learn objectivity is a belief in objectivity, and both are wrong.

Race is not a red herring; even though race may be only a social construction with no objective science behind it, the fact that people believe in and act as if race were real is a problem. In particular it often violates the Civil Rights Acts.

Suppose racism, discrimination and bullying by the general public were the dominant cause of recidivism.. Would it make sense to allow people from the majority regularly do 3 month sentences and keep members of any minority group indefinitely on their first incident?

How do you evaluate how successful an algorithm to decide if someone should get bail is?

The problem with most of these deep learning/AI systems is that they are more or less pattern detection systems. To an AI racial biases in its input data is just another pattern.

Measure against recidivism and skipping bail?

Surely we have a baseline that can be used, such as when these systems are not in play?

If skipping bail is the criteria then how do you detect when you incorrectly denied bail?

Doesn't the verdict of the trial related to the bail request have impact upon that? (legitimate question. I don't know if non-guilty parties are more/less likely to skip than guilty parties?)

That's assuming that we can agree that the justice system works impartially to begin with. I don't know of many people (including myself) that could agree with that.

Use a trial period where you grant bail to everyone (non violent offences only). You could run many different algorithms and models, recording all of their predictions. Since you granted everyone bail, you have a clear picture for which predictors accurately denied bail to the skippers.

> If the software is accurate at picking choices that match the set goals

> auditing of the results vs the desired outcomes

The core problem is this assumption that the goal is to prioritize accuracy. When making "life-changing decisions ... in areas from job applications to immigration", many different goals need to be balanced. Who gets to define what "desired outcome" means?

We already have a hard time simply defining those goals. How would you even begin to define a cost function for e.g. "immigration" decisions when simply bringing up the topic is one of the easier ways to start a heated political argument?

> then it would seem that: the software is doing its job and is 'blameless'

The software is doing exactly what it was told to do: find a particular signal in some set of data (without explicit race. The problem is we spent several hundred years systemically embedding race into most aspects of society, which becomes a minefields of problematic local minima/maxima traps for your ML algorithm.

Let's say we have an good ML algorithm that provides some sort of evaluation about job applicants. Nothing related to race is included. The algorithm is used by a company in Chicago[1][3] or D.C.[2][3] hoping to hire the "best" people. Segregated populations means different schools/etc. Even if you didn't give the algorithm race data, most ML is practically guaranteed to find one of the many ways to infer race. Unintentionally, the ML algorithm ends up doing something similar to redlining[4], because that's what is in the data: the systemic effects of historical racism that still reverberate through society.

[1] https://www.washingtonpost.com/graphics/2018/national/segreg...

[2] https://www.washingtonpost.com/graphics/2018/national/segreg...

[3] red-dots="White", blue-dots="Black", yellow-dots="Hispanic", etc. See the full article for many more examples: https://www.washingtonpost.com/graphics/2018/national/segreg...

[4] https://en.wikipedia.org/wiki/Redlining