> but Amazon says it recommends at least a 95 percent threshold for law enforcement applications where a false ID might have more significant consequences.

That's not in https://aws.amazon.com/rekognition/faqs/ - while that threshold may minimize false positives, I'm curious if it was in any documentation that the ACLU saw. Or, to the bigger point, is it in any public documentation?

Threshold is a configuration property no different than a dynamo read capacity. Misconfigured properties don’t work the way you expect. I don’t see how documentation can be expected to give black and white guidance to what this threshold should be set to without having more understanding of your use case.

> I don’t see how documentation can be expected to give black and white guidance to what this threshold should be set to without having more understanding of your use case.

It can say exactly what the Amazon response to the Verge dismissing the ACLU test should, and, in fact, it should say that if Amazon really does have such critical, specific domain-specific recommended settings, rather than it being something made up off the cuff to mitigate bad press.

You're unquestionably correct. Amazon should spell out in clear, simple, easily accessible language what level of certainty is appropriate for what type of application, giving concrete examples at every point.

With that said, it's possible that no amount of clear guidance would have mattered here. The ACLU isn't exactly asking for better docs.

If you're someone with minimal understanding of the use and deployment of technologies, who knows only of this technology as something used to detect faces, then it's a glaring oversight.

If you're a researcher making a deliberate choice to use the defaults to simulate a random police officer with no real understanding of their tools, then say as much. And compare with more cautious results.

I would be shocked if LE started using this tech, imprisoned a bunch of people, and used the excuse "Well, we followed the FAQS." I would imagine they would work very closely with Amazon and run a ton of calibration. For the ACLU to say that we used the default settings and it doesn't work correctly is disingenuous.

I would not be shocked at all if some LE somewhere used the default settings. It seems absurd to assume otherwise. It's not as though LE has a great track record.

https://en.wikipedia.org/wiki/No_Fly_List#False_positives

Assuming that law enforcement is going to spend any amount of time being careful about enforcing correct use of forensic tools is hard to reconcile with the history of the American judicial system. For example, there is essentially no scientific evidence that a polygraph test measures anything at all, but polygraph evidence is considered admissible in many courts and used to be admitted in many more. Moreover, rigorous studies of fingerprint evidence have found them to have false positive rates as high as 1 in 18, but they are still often treated as close to infallible in court. I am not sure why you think facial recognition is going to be different.

LE doesn't have the best track record following best practices. For example, they still rely heavily on notoriously unreliable eye-witness accounts and polygraph tests, two things science has largely debunked.

They were slow to let go of phrenology as well.

You are going to be very very shocked my friend!

Thinking of an Onion headline: "This tech sucks" says person trying to use niche technology for the first time without knowing its specificities and limitations.

I guess the question is: do we trust any operator to know the specificities and limitations? I for one wouldn't trust Joe Cop to use something like this with the degree of discretion or understanding of statistical nuance called for here.

Would that mean 5% could be false positives? That would be a lot.

From the AWS docs:

> A confidence score is a number between 0 and 100 that indicates the probability that a given prediction is correct. In the tropical beach example, if the object and scene detection process returns a confidence score of 99 for the label ‘Water’ and 35 for the label ‘Palm Tree’, then it is more likely that the image contains water but not a palm tree.

> Applications that are very sensitive to detection errors (false positives) should discard results associated with confidence scores below a certain threshold. The optimum threshold depends on the application. In many cases, you will get the best user experience by setting minimum confidence values higher than the default value.

Which I am interpreting as similar to your question, but perhaps should not be understood as a statistical probability.

On the other hand if you want to judge whether it would work on 300 million people with 95% confidence then running on 535 members of congress with 80% confidence might be a good approximation

(AWS employee)

That’s not how these numbers fit together. Using rekognition, you would create a collection of faces based off your image corpus and then ask the service if a provided image has a match against anything in your collection. Lowering your confidence threshold, invariant of your sample size, is by definition going to increase your false positive rate while lowering your false negative rate.

I think it would be worthwhile to compare a mixed test set and see false positives/negatives at various confidence thresholds, because the fear (IMO) is that law enforcement will minimize false negatives at the expense of a huge increase in false positives.

Wow, if that's the case then this article is quite unfair.

They should rerun this at 95% confidence and see how many mismatches they get, I suspect it will be much lower than the 6.4% they have at 80% confidence, thought it might cost more than the $12.33 they were willing to invest in this hitpiece against rekognition.

You think Police Departments won't knock down the threshold just for justification to knock down some perp's door? They've done far worse to get the results they want.

In my opinion, this is where the crux of the misinformed population epidemic resides. Sensational headline that causes mass amounts of doubt, panic, uncertainty, and outrage will get thousands and thousands of retweets and shares, yet the very valid response to it from the accused will not get any traction.