The thing that makes this trickier than it looks is that these features aren't necessarily encoded in a straightforward way. For example, race could be inferred with some loss of accuracy from zip code. Add more variables and it might be inferred with more accuracy.

Machine learning doesn't necessarily do what you want it to do; it probably will "cheat" and use an easier proxy variable, unless you watch out for it.

There is the urban legend about the machine learning program that learned to "recognize" tanks based on time of day in the training set. It probably never happened [1] but it gets the point across.

[1] https://www.gwern.net/Tanks

Ya, I agree with you that it's definitely tricky business. Including features like zipcode in something like a loan payment predictor is a good example of a sloppy proxy for the underlying and relevant features. Especially when certain data becomes hard to resist using, when something like zipcode of residence might have a 90% overlap with income range and be extremely trivial to acquire, whereas getting detailed and accurate income information from an applicant might be much harder. Tricky stuff.

I am not saying that this is the case in your examples, but sometimes these "predictors" create an unfair feedback loop. e.g. Cops patrol majority black neighborhoods more because of higher misdemeanor rate and because misdemeanor is a strong predictor of crime. This higher patrol rate leads to higher arrests (for crimes such as having weed on you that might not have a significant difference among races and neighborhoods). The arrest data is then fed back into the system, leading to even more patrols, essentially creating a forced equilibrium. Whereas if we didn't use the model the system might have naturally evolved to a different state. This is one possibility of why these models might be effective in the short term but not "fair" (i.e. not effective in the long term). But in any case, I think it is obvious that the benefits of a private company (such as the insurance company) are not necessarily aligned with the benefits of the society.

Misdemeanors are not a predictor of crime, they are crime. Further, many types of misdemeanors are quality of life issues that have outsize impacts on the people who live in those neighborhoods. Go to a town hall meeting in a bad neighborhood and see what people complain about. It's often not so much burglaries as the guys who are chronically drinking and breaking bottles on the corner at 2 am. It's not terribly enlightened to criticize the police for going after QOL violations you would never tolerate in your own neighborhood.

As for whether police attention creates feedback loops, we can check by looking at crime victimization surveys, which generally show the same patterns of racial disparities as arrest records. For that matter, the black / white murder gap has been stable at about 6-8:1 for as long as we've been keeping track, and you can't change that by selectively hassling certain people for weed.

I fail to see how crime victimization surveys disprove possibility of feedback loops. You can have higher murder rates among the black population, yet have overall similar misdemeanour rates. You can even have a higher misdemeanour rate among the black population but have a propostionally even higher arrest rates due to these models and their feedback loops. Or maybe I am missing something in your argument? Care to elaborate?

Well, the victimization surveys tell us that for every category of crime that matters, there are large disparities in base rates that (a) predate policing decisions and (b) recommend focusing on black neighborhoods, because that's where the crime is. The idea that higher arrest rates are a self-reinforcing statistical artifact has no support from the data. The notion that "crime is where you look for it" is specifically undercut by victimization surveys as well as by murder, a crime that is largely impossible to conceal from the state indefinitely. If the baseline disparities are already so high to begin with, there's hardly any variance left over to be explained by self-reinforcing patrol strategies.

Put another way, if you took the first-order strategy you'd come up with by looking at the body count, and then layered the most moustache-twirlingly racist jaywalking policy that you could think of on top of it, the two strategies wouldn't look that much different.

That's all before questioning why it is a bad thing for neighborhoods to be policed for misdemeanors. I lived in West Baltimore for years and I can assure you that my neighborhood was, if anything, chronically underpoliced.

So now you have moved the goalpost to:

>> There might be unfair feedback loops, but they would be effectively negligible.

First of all, it is curious to me that you mention the 8x disparity in murder rate but don't mention that the violent crime survey disparity is less than 0.2x. Secondly, you just claim that the policing has nothing to do with people being arrested at different rates but do not offer any evidence.

Here are the actual numbers if you really care about facts. The victimization rates are 20.5% whites vs 24.1% blacks. Less than 18% difference. On the other hand incarceration rates are 0.7% whites vs 4.5% blacks. That is more than 540% difference. This means the blacks are being arrested at about 32 times the rate at which they commit violent crimes. So your claim that:

>> "we can check by looking at crime victimization surveys, which generally show the same patterns of racial disparities as arrest records"

is not only ignorant and false. It is so far from the truth that it is laughable.

I'd love to respond substantively to this post, and in particular to your figures, but hn has threatened to ban me for discussing this topic. I suppose each of us can only make of that what we will. Best wishes!

I would have personally liked to hear your response but I guess it is what it is. Best wishes to you as well!

Fair points.

> If you use data such as repayment history, income, assets, and other metrics highly related to whether or not somebody is going to pay back a loan, and the resulting model does not output equal representation among arbitrary population groups, I fail to see a problem. If the data being used to train the model has absolutely no knowledge of arbitrary group identifiers, such as "gender" or "race," the resulting output quite simply not biased.

Discrimination isn’t solely measured by input but also by outputs.

https://en.wikipedia.org/wiki/Disparate_impact