Why is that obvious? Take the case of criminal predictions. If you're using convictions as ground truth, then you're just mirroring existing societal biases. For instance, in a society with pervasive racism (like, say 1960s America), the more real-world data you give it, the more your model is likely to converge on racism.

Computers aren't racist but to characterise the article as referring to something like that is to fail to understand it.

Not as bleak as you imply. Just don't give the model race or strong indicators of race as an input.

That's the problem. If there is ANY bias shown in your results, your algo (actually you) will be accused of prejudice. That is what happened with the Re-Offending Risk Assessment in the linked article. It didn't have race or anything like it. It had suburb.

Please. Zip code is a well known proxy for race given the history of racist housing practices. Don't believe it? Ask yourself: What does a person from Harlem look like?

Zip code is not a proxy for race and it is idiotic the say that it is. The zip code of Karl Malone is the same as Chuck Norris. Zip code is a proxy for COMMUNITY.

You're trying to shift the words because it's uncomfortable, but the underlying facts remain the same. Community is often dominated, and in fact is defined by, by race. You can cherry pick a millionaire here and there but those are exceptions but it's disingenuous and ahistorical to assert that this isn't true. Redlining existed to maintain racial divisions. Cities, counties, and even states explicitly maintained racial divisions, and even though these laws do not exist today, their effects are maintained.

Do you think it's a coincidence that the only community with a significant number of black people on the peninsula is East Palo Alto? It's not. ( https://techcrunch.com/2015/01/10/east-of-palo-altos-eden/ ) Do you think it's a coincidence that white people live north 8 Mile Road in Detroit? Shouldn't Marshall Mathers had grown up south of it? It's not. I suggest you read up on your history http://www.theatlantic.com/business/archive/2014/05/the-raci...

> If you're using convictions as ground truth, then you're just mirroring existing societal biases.

And if you don't use real life data...? What do you get?