I disagree with this statement,

> However, just from a cursory review of the mugshots, people of color were disproportionately represented in the mugshot database. So it’s not entirely fair to criticize the facial recognition technology for matching more people of color.

No, the point is the disproportionate representation, and it is a fair criticism, because it is a fundamental limiting factor in the use of the technology. You seem to be distinguishing the "technology" from the data source, but that is not possible. The quality of the technology depends on the quality of the data provided to it.

> I believe a curated list of mugshots with certain characteristics would result in a similar representation of mismatches. Nothing I have read about the technology suggests that there are inherit [sic, is inherent] bias.

This bias is in the data, not the algorithm per se. This is quite accepted in many, even most, criticisms of machine learning applications in the social sphere. It comes up a lot for example in NLP models, for example with assumptions on gender for professions.

It is a technology problem because it demands a technological solution, because removing any and all bias by hand-curating datasets is just not a scalable approach.

I feel like we're missing the popular vocabulary to describe some of these recurring problems.

The algorithms reflect and amplify biases in the data. They also go into feedback loops once they affect the reality being represented in data.

Imagine that an algorithm selects individuals at an airport for search. Contraband smugglers are caught. The dataset now includes them, evolving the bias. We have already seen this at play on social media, recommendation engines, fraud detection, etc.

The future is worrying. There are ratchets on technologies such as these. Unless there are extraordinary counter-reactions, they will be "affecting their own datasets" en masse very soon.

The term I've heard for this phenomenon is "mathwashing."

The old chestnut for this is "lies, damned lies, and statistics."

The problem of disproportionate representation is a problem whether you're using technology or not. It's not a new problem or specific to tech. If a large percentage of dangerous interactions a police officer has with an individual fits a certain profile (e.g. the third party was a man), the police officer will likely exhibit a different set of behaviors when dealing with that party. Sometimes it's appropriate, many times it's not. Same is true for judges and others in the system and you'll see stricter sentencing for certain groups. This results can lead to a self-reinforcing cycle of oppression.

The difference is with technology these biases can at least try to be addressed directly and create an objective more fair system.

The major issue is that the tech is not ready. If you reduce the false positive rate by 5% but you facilitate processing 20% more claims (to use some arbitrary numbers) you are still increasing the absolute number of false positives by 14%. With the punitive American justice system, that is 14% more people who will be detained unjustly, and then when they leave they will be denied most opportunities for housing, jobs, and other things needed to participate in society.

Technology can address biases but we need to make sure that they are addressed sufficiently, so that we do not end up causing more problems than we solve.

> If you reduce the false positive rate by 5% but you facilitate processing 20% more claims (to use some arbitrary numbers) you are still increasing the absolute number of false positives by 14%.

This is a really interesting point that I'll keep in the back of my mind, thanks for that! I hadn't even thought it through in such a concrete manner but I think this really hits the nail on the head.

Some of those false positives are laughably similar. I have no doubt an LEO would be convinced they have the right suspect.

I actually think it’s probably better than most people’s naked eye recognition in a crowd.

Yeah, seems to me like the real problem is that people actually have a lot of doppelgangers, and applying facial recognition to the entire population, even if just as accurate as a real person is just not a useful thing to do.

So this is a good application of the birthday paradox actually and the fundamental misunderstanding people have of this is already a real problem in law enforcement.

Take DNA testing. We all know that given two samples the odds of a false positive are incredibly low. So DNA testing is a great too for eliminating people who might otherwise be suspects or further evidence against the guilty.

The problem is that given this DNA database, someone decided "let's find our suspect by looking for a match". With 1 in a billion chance of a random match and 100M samples, your chances of getting a false positive are really high.

The problem is that there are instances where a DNA match alone is used to prosecute or even convict people.

In recent years this problem has gotten worse due to the rise of familial DNA matching. Given two samples, we used to only have the ability to say if they were a match or not. Now we can say how much of a match they are. How much of a partial match is enough? What's more, you may be implicated by the stored DNA is relatives.

Facial recognition is far more imprecise than DNA. So yeah I fully expect this to get abused by prosecutors and law enforcement.

It's something like the "rare disease" paradox in medical testing and sensitivity and specificity.

If I increase the population I test to include any warm body, eventually I'll end up with more false positives than real positives.

If you are looking for one male suspect and comparing them to all the male faces (let's assume it gets gender correct) in the us, with a 99.9999% (six 9s) accurate algorithm you would get something like:

1 true positive 182,499,818 true negatives 182 false positives

Broad scale facial recognition is just an outright stupid thing to do as a sole measure of identification without the use of other information.

Yes, I think in the modern world with these methods and massive data sets being used in law enforcement, medical sector etc, the people working there should undergo a lot of training about how basic Bayesian inference works etc. We are otherwise in a world of hurt. Healthy people being told they have some really serious rare diseases, maybe even starting treatment, innocent people being prosecuted or killed during raids etc.

Yeah... training almost assuredly won't happen sadly. Especially for cops.

I teach this stuff (biomedical engineering statistics) to undergrads.

The hardest thing for them with probability is false positives. conceptually it does not seem to align well with how the human brain works. I've had students literally get so frustrated the cry because they know what the math says but can't accept that you can test positive for a disease and not have it. They know they are wrong but it's just this weird sticky misconception.

I think you nailed it on the head with, it is 'just not a useful thing to do'.

This seems like a clear example for why facial recognition is a technology that is just not 'solved' yet. The appearance of people's faces, especially from similar ethnic backgrounds, is just too similar for a ML model to parse out with any confidence.

I have noticed this in real life. As I get older, I notice it more and more. I'm sure many of you all have too. There are very distinct patterns, or 'buckets', that human faces tend to fall in. I think our brains tend to naturally categorize them accordingly.

It is probably subconscious. I might not be able to articulate it, or put a definite 'name' on a group. But I know I am constantly seeing patterns of faces in public. People I don't know, and have never met, but they remind me of other random people I have seen in public. Or maybe they remind me of a popular celebrity that everyone knows.

Either way, something goes off in my head. I can't help but think to myself, they must have some sort of similar lineage, or genetic background. I subconsciously categorize them into a bucket with others I've seen.

I imagine this is similar to how Rekognition, and other models work. I thought the blog post from the parent commment @bko, is a fantastic example of this. It is actually amazing, when you think about it, that the ML model can match these faces up as well as it does.

To the naked eye, it is clearly not the same person. Rightly so, considering all the images were in the range of 70-80% confidence. But many are remarkably close. I think this illustrates, the concept I am trying to describe. You can notice it, even with the naked eye.

All of this rambling is to say, I agree with Amazon's moratorium on Rekognition.

As impressive as the technology is, it should probably not be used to try pin-point specific individuals yet, or whatever else folks might be erroneously trying to use it for. If we are to trust facial recognition to identify specific individuals, it should probaly be approaching near 100% confidence, and I imagine that level of confidence is a long ways off.

I sorta think about it as "what's the likelihood of someone winning the lotto twice in a year?" Certainly the probability that any specific person wins twice (assuming here these are independent events...) is so ridiculously small. And yet probability at least 1 person somewhere is that lucky is incredibly high!

Now ask if my neighbor looks like any random person on the street, sure, it's small. But that chance we find two people somewhere that look very similar is incredibly high.

I think the interesting question is what is the chance of doppelgangers given population size and genetic similarity of the population.

Sounds like the prosecutor's fallacy to me unless great care is taken to also consider lots of supplementary information when performing a match. (https://en.wikipedia.org/wiki/Prosecutor's_fallacy)

Recommend reading the part about Sally Clark. The last two sentences are an incredibly short and understated horror story.

"Sally Clark was a practising solicitor before the conviction. After her three-year imprisonment she developed a number of serious psychiatric problems including serious alcohol dependency and died in 2007 from acute alcohol poisoning"

It's only not a useful thing to do if you care about innocent people being wrongfully convicted, which is as you look at the US justice system[0] is debatable. From an outside perspective the US justice system seems to care more about the false negative than the false positive rate of conviction.

There is already a whole slew of dubious methods[1] (bite mark analysis, blood splatter analysis, fibre comparison) being used to put people in prison/to death, these things are known to be bullshit, but they are still being used. I really think we as technology-literate people should fight as hard as we can against the introduction of facial recognition in the justice system because once it becomes common place it will be really hard to undo it.

[0]: https://www.innocenceproject.org [1]: https://www.latimes.com/opinion/op-ed/la-oe-humes-forensic-e...

Denmark has a fairly homogeneous population (discounting the much needed immigration) I swear there are basically 5-8 Danish feature sets that sometimes I amuse myself seeing how they are put together.

This of course means that there are doppelgangers or near doppelgangers all over the place. Even funnier, when i first moved back here from the U.S I was staying at a place in which there was this guy who looked exactly like one of my friends from the U.S, even down to facial expressions and mannerisms (maybe that was because he was always extremely stoned).

Yeah but I am not sure it is suggested to be used in any other context than where a human would be comparing faces too. Ie searching for someone in a crowd or cctv, or checking a passport picture against someone at the gate. None of that is burden of proof level of accuracy. But it just allows to better cover a crowd or process more people at the airport.

If you had accuracy similar to a human, and assuming this is an existing procedure humans do use, then of course it's useful -- you're doing the same task cheaper/faster.

Presumably you'd also use the same techniques as used by humans today to narrow down further, like taking into account location, time, etc of match

Doing the same thing but faster and cheaper doesn't necessarily mean it's better. If there's a high cost to a procedure, then the user would need to show the value of checking a candidate in some other way for it to be worth the effort. Making it easier and faster means you're more likely to check against random innocent individuals.

Fast, cheap, good. You can only ever choose two.

Often, you are lucky to get one.

Seems like the equivalent of p-hacking to me. Sure they could apply complex queries to narrow down the suspects in a fair way, but I don't trust them to do that.

> you're doing the same task cheaper/faster

Move fast and break lives.

I agree. It's incredible that the mismatches were only between 70-80%.

For comparison, I did another test where I took one of those youtube videos that shows you the same person's face over ten years. I used the original 12 year old boy as the image I'm using to match. Over 1,300+ images, it had > 70% confidence in all but 4 images (2 had big sunglasses, one the guy was in green-face, and the other one was actually his wife). And this is from a single picture that's ten years old.

The first part I looked at an open-source facial recognition model that did considerably worse.

https://medium.com/ml-everything/how-facial-recognition-work...

Really? They all look very different to me.

I see similarities in most, especially if I take into consideration that one picture could be "age progressed" image.

> Playing tricks using statistics is not new. OJ’s defense lawyer argued that despite OJ being an abusive husband, it is statistically unlikely that he killed his wife.

> > Only one in a thousand abusive husbands eventually murder their wives

> The more pertinent question is what percentage of murdered women were murdered by their abusive ex-husband?

Shouldn't this be "what percentage of murdered women with an abusive ex-husband were murdered by said ex-husband"?

Statistics is only capable of predicting patterns across a sample of data. It is pretty inaccurate when you try to use it to predict a single data point. The Lawyer knew that but also knew a ton of people don’t understand stats

Your remark on the racial bias is what also came to my mind immediately. It would only be fair if the percentage of mugshots that are of people of color in their 25k sample was published.

This isn't about being fair to the algorithm, it's about being fair to the people subject to the algorithm's judgement

Which is a distorted discussion already. Instead of questioning the benefit of the technology for policing on the whole it is already shifted to the issues of bias. In the end the algorithm and the data will of course have bias in some form but that isn't even important at that point.

Yay, we have an algorithm judging people, but is it fair to Canadians? Completely off...

This isn't distorting the discussion at all. From the point of view of the ACLU, new technology should only be adopted by policing if it can be proven to not perpetuate or exacerbate existing problems of bias, which IMO is a totally reasonable position. We don't accept the "move fast, break things" ethos in fields like aviation, for example, and if we only listened to technologists, society would be too cavalier about collateral damage affecting innocent people. One could argue that society is already too cavalier about this.

I disagree because you already anticipate large scale deployment with that argument.

Facial recognition AI products were already being sold to police departments and customs agencies. Large scale deployment is in the process of happening. Maybe not at your specific agency, but the ACLU is a national organization, and in their eyes one additional false positive caused by increased efficiency is one too many.

Shouldn't matter. What we really want is a classifier that does not take skin colour (or ethnicity) into consideration as a feature.

I know how ML works but I believe we should demand better from these technologies than the limits of our own biases.

Right now, this machine learning algorithm is apparently about as smart as a bigot arguing "yea but percentages show that crime is in fact higher among blacks!". It mainly shows how systemic the racism is, that a dumb ML algo picks up on it.

This is not solved by showing the bigot less statistics about black crime, but by showing them how to pull their head out of their ass.

We should expect no less from our ML technologies, otherwise you'll keep running behind the facts, always fixing errors after they have been learned and made.

Yeah that is hard and we have no idea how to approach it. But the alternative appears to be writing computer programs with the reasoning skills of a racist cop.

Does it matter? More than half of the world's population are people of color.

It matters for the ACLU because it's an American organization focusing on American injustices; in the US, 73% of people are white, 12.7 are black, and 17.6% hispanic.

You're using the strawman and whataboutism logical fallacies, but you're not actually making a point.

Your statistics are misleading. You put hispanics in twice, since the non-hispanic white population of the United States is around 60%.

As someone who hasn't explored the service, thanks for taking the time to do this.

The bearded guy `deb_haaland.jpg` being matched with a non-bearded guy is interesting; shaving would totally fool me, but looking at the match, it definitely looks plausible. Good job, Facial Recognition AI!

Deb Haaland is the lady underneath. The bearded guy is Rick Crawford.