And the idea that the tech is more dispassionate and therefore fairer than humans is a step on the road to hell. I guess you never had an anomalous credit or insurance rating yet. (Notably GDPR has a clause for precisely the ability to review automated decision making where it can have a material effect.)

This isn't a sorting algorithm where there is a simple provable solution, this is putting heuristics in a computer.

I'd hedge that a little. Because I think there are a lot of people who are just naive and think that using data that is historically racist will result in racist outcomes.

Not saying that the person creating the algo is racist, but for such a harsh accusation you need enough substantial evidence. That, or we dilute the term.

I like this: https://arxiv.org/abs/1706.09847

Obviously the designer could make this data set racist on purpose, I was just saying that an AI that judges on race might be due to stupidity and not malice. Hanlon's Razor.

[1]Not realizing that there are other parameters that correlate with race