There are several independent meanings to the word bias: (1) statistical bias = a systematic gap between estimation and reality; (2) a prejudicial bias = an action taken against a person/group on immoral grounds; (3) partisan bias = a stake in the outcome.

We expect judges to be unbiased in the (3) sense; businesses in the (2) sense; and scientific research in the (1) sense.

I have seen even professors confuse these in quite horrific ways: claiming eg., esp. that (1) is just the same as (2, 3).

This claim is common and very bad. Statistical bias has nothing to do with (2, 3). A system can be accurate (eg., an accurate record of historical crime) and still produce prejudice (eg., future over policing).

And likewise, prejudice (2) is not simply "a once-true stereotype", ie., a human-equivalent of (1) (a compu sci professor claimed this at a public lecture -- quite insane). This leads to the view that "black people were more criminal" -- no, there are no such thing as "black people".

Ie., the very concepts which prejudicial views partition reality into leads to datasets. Prejudice isnt a statistical effect, because statistics requires a measurement against a set of concepts (races, distances, etc.).

In (2) the term 'immoral' is doing wizardly levels of work.

These days in the context of 'social bias' the word 'bias' seems to mean 'difference between what happens and what I think is fair'.

Yes, deliberately so. Whether an action is prejudicial is not a statistical question, but an ethical one. The temperature of frozen water is likewise, not a statistical question, but a scientific one. Both questions "require a theory".

Statistics itself is essentially pre/post-theoretical analysis -- theory has to be added to statistics.

My use of "immoral" here is to sign-post that something is missing: an ethical theory.

I have seen esteemed people try to reduce (2) to (1) -- you may as well do so with scientific questions, ie., eliminate all of chemistry in favour of just "how often ice melts at a given temperature". We know, of course, that we do not even have data for the whole range of actually knowable states of water -- because we have theories which describe them. (And without theory, we'd not even know what data to have collected.)

Likewise, ethics is not a matter of outcomes being different whenever they are different. Ethics, like science, is a discrete constraint on theory-building -- not on data analysis.

> Whether an action is prejudicial is not a statistical question, but an ethical one.

I think this is kind of what we're talking about, though, isn't it? Does "prejudicial" mean "having preconceptions about a person based on a few observed characteristics"? Or does it mean "assuming things I think are unfair about a person"?

> My use of "immoral" here is to sign-post that something is missing: an ethical theory.

If that was the intent then it worked for me.

> This leads to the view that "black people were more criminal" -- no, there are no such thing as "black people".

I am confused by this sentence. If there is no such thing as black people — and I can understand the reasoning that would lead to this conclusion, — then what is meant by the daily application of the word (black students, black Americans, black and brown community, etc.)

There is such a thing as ancestry (coming from a similar historical geographical location) which leads to common features (eg., skin color).

However, eg., if you take all people labelled "black" around the world, you'd find as different ancestries between them as-if you had taken "black and white". ie., the label "black" isn't a particular ancestry, nor even similar ones.

"Black" is something invented somewhat recently (early modern period) which was initially a hypothesis about there being human subspecies and there being such a thing as a "black people". This hypothesis is false, and it really took knowing about evolution in the 19th C., and genetics in the 20th, to see why.

Today then "black" is many things but perhaps best understood as a social caste which has been historically invented and reinforced by state institutions and latterly by culture.

It's a bit like "working class" and a bit like "tall people" and a bit like "a nationality". In the sense its blends a political grouping (class) with a physical signifier (skin color), and it is something its people identify with ("I am X").

The key issue here is that there can be no "racial dataset", in that as a scientific hypothesis races don't exist -- and as an ethical constraint, we ought never act as-if they did.

So we have these dual constraints on our theory-building and on our data collection: a scientific one, and an ethical one.

These are not constraints that can ever be deduced (induced etc.) from a merely statistical analysis of data -- and thus (2) is not (1).

(Consider, eg., that by the state creating and reinforcing a "black race" it induces commonalities between people labelled by that category. So if you actually collect data on "black vs. white" in any country, you will find "the categories you are looking for" -- precisely because youve induced these differences. It is in this sense statistics alone cannot tell you anything.)

One of the problems with this article is that it would prefer the algo do something else than truthfully represent the training data. If you really want an affirmative action policy for admittance then you need to adjust the score only after the algorithm had its say in predicting outcomes.

Bad things happen if you require an AI to lie.

“This situation is in conflict with the basic purpose of HAL’s design – the accurate processing of information without distortion or concealment. He became trapped. […] HAL was told to lie. But people find it easy to lie. HAL doesn’t know how. So he couldn’t function. He became paranoid.”

I wonder if this is algorithm bias or teaching material bias? I can't see anyone writing "if name not in white_male_names then score -=15", but take successful profiles, generalize and create predictive model.

Anyways, this should be pretty easy to test: Just rename applicants with pseudonyms before rating.

The algorithms being only as good as we are has many levels to it. Encoding existing biases is one. But nowadays it’s not existing algorithms but rather data driving the decisions. So the biases in how data might be collected or processed is at least the second and much more difficult to discern layer. Finally, what if we find the data to be correct, the AI cost function impeccable and still find a bias? I.e. what if we find there’s a bias in the larger system of society? At some point the entire exercise becomes futile, as the passionless machine cannot be made to fit perfectly into an imperfect society where we’d like to make decisions not because they make sense but because we feel them to be right. Let’s keep the machines where they belong, shall we?

Also, come to think of it, a machine cannot (at least for now) be taught about detecting and avoiding feedback loops. Data might suggest an ethnicity A to be more likely to be criminal which leads them to be filtered out for spot checks more frequently which enforces the algorithm.

I can't see how this would cause a feedback loop. You can still check the data to see what percentage of each ethnicity has been checked and found to be breaking the law (compared to checked and not breaking the law) and use that as a guide on which one is more criminal. Of course, you shouldn't be able to set an ethnicity to zero otherwise they would never be checked again.

To give a trivial but excessively stupid example of a feedback loop imagine a delusion "Redheads are murderous bandits - kill them on sight!".

Under that barbaric policy held as dogma the only redheads likely to survive would either avoid them entirely (and not be factored into the data) or in turn start killing in the crazed non-redheads in self defense, and being vastly outresourced looting as a matter of survival. The idiotic prophecy turned self-fuffilling as reflected by the data.

The entire premise of checking by ethnicity is also incredibly dumb - as an enforcer you shouldn't care what they look like you care if they are breaking the fucking law or not! If you lack any causology in your examinations what you have is automated superstition.

I think it can be done. Just maintain the data not in a standard tabular format but a time based graph instead. And put constraints to not look back. Constrained deep learning is a thing too.

The issue is that the machine has no causal model of how the predictions are leading to the data. You could, as you say, try to single-out some variables and rig how they're processed.

Here I dont think that helps: so long as you predict area A has more crime, area A is more policed and thus always appears to have more crime.

The issue, in my view, is not the data nor the algorithm -- no modification to either can fix the issue. The issue is the machine isn't embedded in the world, and esp. has no ability to acquire a rich understanding of the human social environment.

A fundamental aspect of understanding X is knowing what is irrelevant to X, ie., what "data" it is permissible/essential to ignore. In the case of crime, one ought ignore data from over-policing -- but this is not an effect which is present in the data itself.

> But nowadays it’s not existing algorithms but rather data driving the decisions.

There are different flavours of Machine Learning algorithms. You are referring to one of those where the learned representations are created from raw data. There are other flavours where algorithm designers manually input the important aspects of the representation. For example, a popular credit scoring agency where I live - uses the postcode as an input. The postcode in some cities is directly correlated to race and ethnicity.

The broader point is that there exists various aspects that drive bias and that goes beyond data.

It is entirely obvious that biased results are the results of real-world biases, at least in most cases. That explains them, but it doesn't excuse very much. Reproducing, for example, incorrect hiring or credit decisions is wrong independent of the causality, and harms both those individuals and society (yes, banks lose money when they turn down perfectly good customers for irrelevant reasons)

One misunderstanding in this debate is that AI researchers get very defensive and retreat to, basically, "I'm not racist". What they miss was that (in most cases) they were never accused of anything like that. Everybody understands how these biases arise from the data, and are sometimes amplified by it.

What they can sometimes be accused of is not checking for the most obvious flaws in their models before publishing them. At this point, the problem is well-known and there's simply no excuse for not checking. And if that means your model cannot be released because you don't have the data, that's too bad, but your work just isn't up to purpose. Go find better data. We don't let people sell rotten steaks just because the stores are out of the fresh variety, and there's no reason to treat AI differently.

The installment of the series linked above doesn't discuss machine learning at all. It just discusses a rule-set hand-coded with prejudicial bias. (-15 points for non-European name, -3 points for women...)

In other words, one guy who preferred Caucasian men as candidates for the medical school wrote the code with his preferences deliberately baked in. This was so obviously bad they got caught, sued, and ruled against.