If we included every aspect about a person in some statistical model, we may discover "uncomfortable truths" that hold true for the general population. But these truths, while statistically correct, may fail our test for what we consider to be philosophically fair, and ultimately undermine an individual's agency to act independently.
So perhaps in your experiment, the problem is that our feature selection is not reflective of the values we'd like to uphold, and that aspects like "had lead poisoning as a child" is not a sound feature to include in our model because it measures aspects of a person outside their control. Instead maybe our feature set should only include aspects that measure facets that are under the individual's control such as community service, whether they still associate with other criminals, whether they have or are pursuing education, whether they have children to care for, etc. (or some other feature set that's more thought out and sound, but you get the gist)
This still may not have as good accuracy as a model that included other features about the person, but it's arguable that this system would be more fair, especially over a model using more features but was artificially fudged to satisfy some prior about what we consider fair/unbiased.
This is exactly what the author is talking about. You are comparing the predictions against your fantasy of a world where these aspects do not matter because they're not "fair". When they don't match up, you call the predictions biased. But these factors outside our control do matter, accounting for them does not introduce bias, and averting our eyes will not change that fact.
My suggestion is that we should be using a better feature set that only looks at aspects that we can reasonably hold an individual responsible for rather than using demographic information which is out of an individual's control. If we have two convicted criminals with similar crimes, behaviors, and histories, but one is white and grew up in a wealthy neighborhood while the other is black and grew up in a poorer town, why should the former be granted a higher probability of parole than the latter? Why should either of them be held responsible for the actions of others? Even if in expectation people from the latter demographic were more likely to reoffend than the former, that is not justice – it undermines liberty.
So my philosophy of "fairness" here is that we should hold people responsible only for their own actions, which, while off-the-cuff, seems like it would agree with most justice systems. If we included demographic information in our models, we would effectively be holding individuals responsible for the actions of others, which doesn't seem sound.
This is true of all predictive variables and all predictive models. It doesn't justify the recent panic aimed at ML at all.
I think the so-called panic comes from the fact that more complicated algorithms are difficult for an untrained person to understand. For example, it's also illegal to discriminate housing/lending based on race, but if I hide that discrimination inside of a sufficiently-complicated ML model, I may be able to get away with it.
The gist of the article is that statistical bias is not the bias that journalists are interested in. The article doesn't discuss how these biases are related or relevant to each other, but rather assumes that statistical bias is the only one that should matter. I think the article is missing a discussion of the gaps between what are the measured inputs into your statistical model, and what can be acted upon from a policy perspective.
As a thought experiment, suppose the only two inputs that determine a person's recidivism rate is their past criminal history and whether they had lead poisoning as a child, but that of these two we can only measure past criminal history as an input into our algorithm. If race is strongly associated with childhood lead poisoning (such as in real life [1]), then our algorithm might get higher classification accuracy by including race [2] as an input in its training data. This might have less statistical bias, but would bias against individuals of a race who are in truth not at higher risk of recidivism.
[1] https://scholar.harvard.edu/files/alixwinter/files/sampson_w...
[2] The actual COMPAS algorithm doesn't use race as an explicit input, but that doesn't really change the issue.