For example imagine some ML algorithm that predicts the potential value of a customer (or their likelihood to steal from you) as they walk in the door of your business. How do you remove the racial bias from a system like that when racial biases exist in society that result in the average White person being in a higher socioeconomic class than the average Black person? The ML algorithm will tell you White customers are more valuable than Black customers. It is doing exactly what you told it to do, but the end result is that Black people get worse service in your business.
It's saying that the dataset isn't reflective of the world, and this is an issue because then you'll end up with voice tools[1], facial recognition software[2], and sentencing algorithms[3] end up performing disproportionately poorly on some minority groups. And this ends up reinforcing the poor positions many people are already in.
1) https://www.pnas.org/content/117/14/7684
2) http://proceedings.mlr.press/v81/buolamwini18a.html?mod=arti...
3) https://www.liebertpub.com/doi/full/10.1089/big.2016.0047
Well, that's not a very full story. It was scientifically invalid because it modeled mental traits based on the shape of the skull, and that's the important part. If you trained a model to predict traits based on accurate data about skull shape and traits, I'm confident that the algorithm would happily do that. Phrenology would still be bad regardless of what technical methods are used to do phrenology.
I really don't want my field to regress towards 19th-century pseudoscience when we can empower other scientific advancements that will yield immediate benefits.
This is a strange point. Are personality types subjective? Is IQ? Maybe they are, but we still sometimes find them useful to quantify. If it's ok to predict these things from your answers to questions...why isn't it ok to predict them from pictures of your face?
So yes, I think I can state pretty concretely that there's no value in trying to predict IQ from face shape.
The presence of error in a prediction makes that prediction valueless?
Can you describe a situation where the harm caused to someone by mis-classifying their IQ is outweighed by the increased efficiency from...being able to more quickly identify someone's IQ? Like, can you even describe a situation where this will be used ethically at all? What use is there for predicting someone's IQ with dubious accuracy, assuming you already have their permission?
Yes.
> Do you think their error rate is zero?
Well it depends. Do I think the error rate of IQ as a predictor of underlying intelligence is flawed? Yes.
Do I think that IQ as a measure of IQ is flawed? Still, often, yes.
> These things don't need to be perfect to be good.
To my point above, I generally am not a fan of IQ in general. Psychology researchers are consistently finding new and interesting ways that IQ tests are socially/culturally/environmentally influenced and that exams may not be fair.
If your claim is that you can somehow train an algorithm to look at someone's face and evaluate their underlying intelligence better than the tools we have today, honestly "delusional" is the first word that comes to mind. That's just not something that currently available tools can do. And thought experiments about a "perfect" classifier aren't interesting when we're discussing applied ethics.
And that comes with the huge, enormous, gigantic, asterisk that you can even measure "intelligence". Like, it comes with the caveat that you can even reasonably define intelligence. We generally measure intelligence as correlated with success at some metric: math problems, visual puzzles, chess, reading comprehension, whatever. If you're contending that a hypothetical model could predict underlying intelligence better than the measures we have today, how would we even know? When training a model you have an evaluation function.
If the evaluation function is flawed (which is essentially your contention, and which I absolutely agree with), the trained model will exhibit biases that reflect the flaws of the evaluation function. An ML model isn't suddenly going to solve the cultural issues we have with measuring IQ. It will encode the same biases that society does, because the model will try as hard as it can to do exactly what the human proctors would have done.
These are exactly the kinds of broader ethical questions that Timnit points out we need to worry about.
> If the evaluation function is flawed (which is essentially your contention, and which I absolutely agree with), the trained model will exhibit biases that reflect the flaws of the evaluation function. An ML model isn't suddenly going to solve the cultural issues we have with measuring IQ. It will encode the same biases that society does, because the model will try as hard as it can to do exactly what the human proctors would have done.
Of course. But so what? There are people using IQ now for things. An ML model isn't going to magically make those biases worse, either. What it's going to do is bring them to the surface, so that they are quantifiable and we can actually do something about them.
ML is the solution to the bias problem. Right now these evaluations are being made by other humans. Humans who we cannot statistically debias. Humans who's biases we can't even effectively interrogate. The reason people are making all these memes about biased AI is not that it's more biased than humans, it's that the bias is more measurable.
Well, actually, research shows that unless great care is taken it absolutely can. If you include for example race as a factor in a model, it can learn non-causal correlations between race and whatever the objective is. This can have a compounding effect in some cases. [0]
> What it's going to do is bring them to the surface, so that they are quantifiable and we can actually do something about them.
I don't follow this. If we have a biased objective function, the model won't surface any biases we weren't already cognizant of in the objective function. And they were already quantifiable: we had a function that we were using to evaluate the model. We could use that same function on whatever non-model evaluation we were doing.
> ML is the solution to the bias problem.
This is basically directly in contradiction to what leading experts on the subject say. ML cannot fix bias in human systems, unless we presuppose that those systems are biased, in which case we can often address the bias in the human systems directly without ML.
> Humans who we cannot statistically debias. Humans who's biases we can't even effectively interrogate.
You can still have decisions be made by objective expert systems without complex ML. If you want to learn someone's IQ, the best way is to debias the IQ test, not to try and infer it from their face bones.
> it's that the bias is more measurable
If we can measure the bias in the output of an ML model, we can equivalently measure the bias in the output of a human system. You're presupposing the existence of some unbiased objective function which we don't have, and that's at the core of the issue.
[0]: https://www.wired.com/story/ideas-joi-ito-insurance-algorith... has a few good examples here, like how naive bail and sentencing models encode racial bias that isn't present in humans. And to be clear the response here shouldn't be "well let's just build better models" but "why do we think a model will improve the situation here at all"? Removing agency from Judges has historically been bad for the average person convicted of a crime. This doesn't mean that individual judges can't make terrible rulings, but that the alternatives are usually worse on the whole.
We can actually follow the logic of the model. For instance, you can theoretically de-bias a dataset by building a racial classifier from it. What you need is an objective test for the presence of racial information, and that's easy to obtain: Build a classifier to explicitly predict race from your feature set. Train an adversarial model to reconstruct your dataset with maximum fidelity, subject to the constraint that race can no longer be predicted from it.
> This is basically directly in contradiction to what leading experts on the subject say. ML cannot fix bias in human systems, unless we presuppose that those systems are biased, in which case we can often address the bias in the human systems directly without ML.
These experts are just wrong, then. Naive ML won't fix bias in human systems, but that doesn't mean we can't use ML to fix it, if we do so thoughtfully.
> You can still have decisions be made by objective expert systems without complex ML. If you want to learn someone's IQ, the best way is to debias the IQ test, not to try and infer it from their face bones.
Sure, but there are a lot of things that we don't do in the best possible way because it's too expensive. There are lots of use cases for cheap, scalable, low precision models.
> If we can measure the bias in the output of an ML model, we can equivalently measure the bias in the output of a human system. You're presupposing the existence of some unbiased objective function which we don't have, and that's at the core of the issue.
Right, but we cannot fix the bias in a human. And humans are heterogenous and inconsistent. The same person may be more or less biased on different days. The ML model is consistent, and we can incrementally improve its bias in tangible and testable ways. The same is not true of humans.
And what does this get you? Let's look at a face recognition dataset. What happens when you debias it? Is it still useful? No. Because the faces no longer resemble real faces.
> These experts are just wrong, then
Perhaps, but you aren't making a strong case for that.
> There are lots of use cases for cheap, scalable, low precision models.
That involve facial recognition?
> Right, but we cannot fix the bias in a human
We don't need to. We just need to fix the bias in the system. And we absolutely can incrementally reduce bias in systems that involve humans.
Not to you. But you can remove the racial information without destroying all the information that a model can detect.
That's what the ethicists say: don't use facial recognition models. Don't work on them. Don't research them. They cannot be both unbiased and useful. And in general, there's few to no uses that are ethical, period.
Some of these correlate with race. But only part of the information correlates with race, not all of it. It is, in principle, possible to remove the information that identifies race without destroying the information that identifies the individual. It is true that part of an individual's essential characteristics are their racial characteristics, but it is not true that the only way to identify an individual is their racial characteristics. For instance, there is no way that i'm aware of to infer race from fingerprints, but you can absolutely identify a person by their fingerprints. So, the question is, can we extract a facial fingerprint that identifies a person, but not their race? I think the answer is almost certainly yes, and it is going to be up to a clever model design to do it. But essentially it would look like a GAN where the adversarial component is constantly trying to predict race, while the Generative component is trying to trick the race classifier without tricking the person-identifier.
Or perhaps they understand that this won't work in practice.
Personality, aptitude, criminality, intelligence, empathy, etcetera... that's almost surely pseudoscience.
> We have noticed a lot of concern that PULSE will be used to identify individuals whose faces have been blurred out. We want to emphasize that this is impossible - PULSE makes imaginary faces of people who do not exist, which should not be confused for real people. It will not help identify or reconstruct the original image.
This is just another way to generate realistic faces that match a pattern. The "bad use cases" I can imagine involve unmasking real people, which is not what the model does.
For example, there are already attempts to use expert systems in the courts - to "scientifically" determine flight risk (and thus bail), and even chance of re-offence (and thus punishments). Those are trained on datasets of convictions. But we know that both conviction rates, and arrest rates, are themselves indicative of bias - e.g. some people get arrested more often for a crime, for which other people might get a verbal warning from police.
So ML takes those biases, and entrenches them further. Worse yet, because people tend to believe that "computers cannot be biased", the fact that it's ML that's producing a specific result, is taken as prima facie evidence that said result is immune to criticism from that perspective. And with most advanced models, you can't really crack it open and make sense of what's inside - so you can't easily prove that it is biased.
I can think of at least one trick that might work. Build a classifier capable of identifying race from the data, and then reverse its gradient to remove its signals. This would deracialize a dataset, so that there is no detectable latent information in it.
However, whether or not my particular solution works is somewhat beside the point. It's an engineering challenge to figure out how to do this, but that isn't an argument that ML is fundamentally bad for these purposes.
> For example, there are already attempts to use expert systems in the courts - to "scientifically" determine flight risk (and thus bail), and even chance of re-offence (and thus punishments). Those are trained on datasets of convictions. But we know that both conviction rates, and arrest rates, are themselves indicative of bias - e.g. some people get arrested more often for a crime, for which other people might get a verbal warning from police.
I totally agree that this is a problem. But it seems to me like the solution is more effort on fixing the biases, not reverting to human judgment. I think it's always important to make sure we understand what the baseline is we're comparing to. Humans are racist. Humans created the biases in these datasets. These algorithmic tools just bring those biases to the surface and make them quantifiable. It seems to me that that represents an incredible opportunity to actually fix the biases in a measurable way.
To be clear, I do get arguments that ML might increase the efficiency of policing, and there is some inherent tension between the efficiency of the state and freedom, to the extent that imperfect enforcement is a tacit feature of our legal system. But, this is not an inherently racial point.