Under that barbaric policy held as dogma the only redheads likely to survive would either avoid them entirely (and not be factored into the data) or in turn start killing in the crazed non-redheads in self defense, and being vastly outresourced looting as a matter of survival. The idiotic prophecy turned self-fuffilling as reflected by the data.

The entire premise of checking by ethnicity is also incredibly dumb - as an enforcer you shouldn't care what they look like you care if they are breaking the fucking law or not! If you lack any causology in your examinations what you have is automated superstition.

Here I dont think that helps: so long as you predict area A has more crime, area A is more policed and thus always appears to have more crime.

The issue, in my view, is not the data nor the algorithm -- no modification to either can fix the issue. The issue is the machine isn't embedded in the world, and esp. has no ability to acquire a rich understanding of the human social environment.

A fundamental aspect of understanding X is knowing what is irrelevant to X, ie., what "data" it is permissible/essential to ignore. In the case of crime, one ought ignore data from over-policing -- but this is not an effect which is present in the data itself.

There are different flavours of Machine Learning algorithms. You are referring to one of those where the learned representations are created from raw data. There are other flavours where algorithm designers manually input the important aspects of the representation. For example, a popular credit scoring agency where I live - uses the postcode as an input. The postcode in some cities is directly correlated to race and ethnicity.

The broader point is that there exists various aspects that drive bias and that goes beyond data.

One misunderstanding in this debate is that AI researchers get very defensive and retreat to, basically, "I'm not racist". What they miss was that (in most cases) they were never accused of anything like that. Everybody understands how these biases arise from the data, and are sometimes amplified by it.

What they can sometimes be accused of is not checking for the most obvious flaws in their models before publishing them. At this point, the problem is well-known and there's simply no excuse for not checking. And if that means your model cannot be released because you don't have the data, that's too bad, but your work just isn't up to purpose. Go find better data. We don't let people sell rotten steaks just because the stores are out of the fresh variety, and there's no reason to treat AI differently.