I am from Brazil.

On the first day of trials of deep-learning based facial recognition here, a random person was arrested because the algorithm confused that person with another one.

Even more stupid, is that the person with "outstanding warrant" was actually ALREADY in prison.

So yes, AI managed to arrest the same person, twice, one time the real person, one time a random look-alike.

Seems like a different situation - the facial recognition algo was wrong. Not the same as an AI resolving a face that resembles someone and then prosecuting that person on the basis of the image.

You realize that the idea is the same, right? AI made an incorrect determination and people ran with it.

You do understand the difference between arrest and prosecution? People are arrested on the basis of a weak mistaken identity all the time, no AI needed

I remember that time a guy's life got turned upside down because his fingerprints "matched" those found on a bomb. Despite the fact that he had no motive, disposition, or access - they were determined to convict, because to do otherwise would be to admit that fingerprint analysis doesn't enjoy the scientific foundation that DNA evidence does.

So yeah, real people have been harmed by bad matching algorithms.

Yes, it is: https://news.ycombinator.com/item?id=26504144

Ok that’s pretty far removed from a prosecutor relying on that image though

Imagine low-resolution face photo. Police found 3 men whose faces are similar to that photo. Now they're super-resolutioning that photo and suddenly first person looks very similar to it, while second and third persons do not. First person is in danger because of algorithm which was trained on some specific photos.

One can imagine a lot of future scenarios. In fact that describes is an entire genre of story writing known as “science fiction”

It isn't "science fiction" though, we have a mountain of examples that demonstrate how probabilistic AI fails when the training dataset doesn't perfectly align with the eventual application. It isn't restricted to AI either, there have been numerous studies showing that cross-race identification to be garbage data.

<Insert other race> all look the same: https://onlinelibrary.wiley.com/doi/abs/10.1002/acp.898

This bus is an ostrich: https://arxiv.org/abs/1312.6199

Racist autofocus: https://sitn.hms.harvard.edu/flash/2020/racial-discriminatio...

Google thinks black people look like gorillas: https://www.wired.com/story/when-it-comes-to-gorillas-google...

And here is an argument to look forward to: should the training dataset be build to represent the general population, or the specific subgroup that the software would be most often encountering? Because that FBI UCR...

Out of curiosity, do you think forensic science methods in general are likely to randomly incriminate people?