News for hackers. Stuff that matters.
Re:Thoughts On Machine Learning Accuracy by komali2 on Friday July 27, 2018 @03:10PM (#17628382)
I'm not only disappointed in the ACLU for misinterpreting the results, but also because I usually have a lot of faith in their competency, and they absolutely should have seen this response coming from Amazon.

What'd they think Amazon was going to do, roll over and be like "turns our our facial recognition software is racist, woops!"? Now instead of a meaningful dialog, we've got a line in the sand where on one side, the ACLU, champion of the people's rights, doesn't understand technology, and on the other side, facial recognition software is Bad and Evil. Shits so polarizing these days it seems there's no room for negotiation, as much as I'd like there to be.

Re:Thoughts On Machine Learning Accuracy by confounded on Friday July 27, 2018 @04:12PM (#17628934)
> I'm not only disappointed in the ACLU for misinterpreting the results...

Did they? This blog post describes some implementation choices which could make their false positive rate lower.

It reveals no information about how Rekognition is being deployed by LEAs, and there’s no meaningful regulation or oversight about how that happens.

We can’t tell what the assumptions and implementation choices of local police departments are, because that’s top secret information. Most people acting on the predictions have no idea what they are, let alone the implications.

It’s fine to ding the ACLU study on methods —- you can, because they published some.

No-one would argue that a toy model based on members of Congress is representative of the public at large. But it’s more representative that a press release from a company that are trying to normalize and monetize mass surveillance via facial recognition.

Re:Thoughts On Machine Learning Accuracy by l9k on Friday July 27, 2018 @04:17PM (#17628985)
The original ACLU title is: Amazon’s Face Recognition Falsely Matched 28 Members of Congress With Mugshots. It was spread on the news with the same conclusion.

It puts directly the blame on Amazon and its technology and doesn't mention any configuration they used.

Re:Thoughts On Machine Learning Accuracy by confounded on Friday July 27, 2018 @04:32PM (#17629100)
Why would a title mention API/service configuration? This is not common practice, even in academic applied ML papers.

They used the default settings, and a training set which seems extremely likely to be used in ‘LEA-production’

Any user, including LEAs, is free to use which ever configuration options they like.

The criticism with the most statistical implications is the confidence level used. The ACLU were clear they they used the defaults. I can take at face value that moving from 80% to 99% confidence on a sample of 500 faces could produce 0 false positives. However, on the faces of the tens/hundreds of thousands of people that might move through the center of town on any given day, the implications for causing ordinary people serious harm are large.

I am personally shocked that Amazon are willing to “recommend” law enforcement actions at any level of confidence, from an unintrospectable machine learning system.

Re:Thoughts On Machine Learning Accuracy by stevew20 on Friday July 27, 2018 @08:03PM (#17630548)
I personally believe that facial recognition software will eventually be employed by law enforcement no matter what we do; I also believe that this is a bad thing and lies on the path to a controlling and pervasive police state.

However, this article states clearly that there are recommendations to LE on how to use Rekognition in a meaningful and correct manner for LE's use case. I am highly confident that Amazon will provide technical documentation and manuals for a contract of that size; of course Amazon will not be able to conduct oversight on LE operations. LE may missuse it, the way they've missused Stingrays, stoplight cameras, clipper chips, and other devices.

You state that having 99% accuracy for identifying an individual will do large scale harm, and I agree; what about the 1 person out of every 100 that is falsly identified? The other side to this coin is the number of people law enforcement would miss recognizing without this software... This argument comes down to human ability vs. machine learning ability, and human discretion in interpreting results. I think with proper use it'll be a benefit to identying culprits quickly.

Regardless, it is still pretty evil giving LE any tool that can autonomously monitor people who have not committed any crimes. But who's going to stop them?

Re:Thoughts On Machine Learning Accuracy by cptskippy on Friday July 27, 2018 @04:52PM (#17629293)
The title is akin to saying "We bought <gun maker>'s hunting rifle, pointed it at Congress, pulled the trigger, and it shot 22 members of Congress". It's a tool, you're responsible for how it is used, not the manufacturer.

I would imagine the confidence level would be tweaked by Law Enforcement or anyone else based on the results. If you're monitoring cameras for missing persons or shooting suspects and getting 0 hits, you might lower it because it's better to dismiss false positives than the alternative. Conversely you might increase it if you're overwhelmed by the number of false positives.

Re:Thoughts On Machine Learning Accuracy by d4l3k on Friday July 27, 2018 @05:25PM (#17629540)
Since you're comparing facial recognition to guns and guns are regulated by law and have strict rules for use, perhaps facial recognition should as well?

I think there should be a discussion about laws/regulations about facial recognition use. Even if Amazon removes their service there will always be companies that don't care about the ethics as much. Strict rules instead of public shaming seems like the better approach

Re:Thoughts On Machine Learning Accuracy by RhodesianHunter on Friday July 27, 2018 @11:41PM (#17631389)
If you screw up with a gun someone dies.

If you screw up with facial recognition a human double-checks and nothing happens.

Scissors are dangerous but we don't regulate those either.

Re:Thoughts On Machine Learning Accuracy by syshum on Saturday July 28, 2018 @06:41AM (#17632549)
ummm no...

First of it is not a given if you "screw up with a gun someone dies" that is not the only result

Further it is laughable to believe that with facial recognition a "human double-checks" we have seen time and time again that new technology entering the realm of criminal justice can be used to wrongfully convict people at an alarming rate. See Hair and Bite mark controversy or the number of Labs that have been compromised in recent years

The fact is that a "humans" that are suppose to be "double checking" do not in reality, if the computer spits out "Well the computer says it is person X" that is what the Jury will hear, and the Jury will not question the technology and the person will end up in prison until such time as someone like the Innocent project comes along to invalidate either the technology on the whole or the application of the technology

Re:Thoughts On Machine Learning Accuracy by sangnoir on Saturday July 28, 2018 @11:54AM (#17633660)
> If you screw up with facial recognition a human double-checks and nothing happens.

You have too much faith in humanity: a lot can be, and will be done "because the system says so". Nevermind integrators who will create processes that assume recognition is 100% correct "Yes, I know you're not the wanted criminal, but now that I have tagged you, I have to take you to the station and process you out...Monday morning because that's when Jim is in"

Re:Thoughts On Machine Learning Accuracy by confounded on Friday July 27, 2018 @05:28PM (#17629562)
> The title is akin to saying "We bought <gun maker>'s hunting rifle, pointed it at Congress, pulled the trigger, and it shot 22 members of Congress".

Only if <gun maker> was specifically marketing their hunting rifle to be aimed at members of the public, and for the trigger to be frequently pulled. Happily, gun-makers are far more responsible than that.

> It's a tool, you're responsible for how it is used, not the manufacturer

Not so. Private businesses can refuse to offer services largely as they wish, and AWS frequently do, based on their terms of service.

For example, you are not permitted to run a web-scraper or open mail relay on EC2. You are not permitted to host bestiality on S3. None of these things are illegal. But Amazon have taken the stance that they just don't want them to happen on AWS.

I am of the view that mass facial recognition poses more problems for society than running web-scrapers.

Many civil liberties organizations have serious concerns about the use of facial recognition for law enforcement, including pro-industry organizations such as the EFF[0]. This is not some new reaction; they have had Know Your Customer guidelines about terms for ML for states since 2011[1].

Even competitors who are incentivized to sell facial recognition tech to LEAs are warning that it's a bad idea for society, with serious and likely lethal implications. Here's the CEO of facial recognition startup Kairos[2].

> ...the confidence level would be tweaked by Law Enforcement

This is a large part of the problem. If you don't think so, then we simply have very different visions of the future that we want to live in, and must agree to disagree.

[0] https://www.eff.org/wp/law-enforcement-use-face-recognition

[1] https://www.eff.org/deeplinks/2011/10/it%E2%80%99s-time-know...

[2] https://techcrunch.com/2018/06/25/facial-recognition-softwar...

Re:Thoughts On Machine Learning Accuracy by joshgel on Friday July 27, 2018 @06:46PM (#17630073)
> Happily, gun-makers are far more responsible than that.

I mean, accurate, but then again they are making guns, so ... low bar i guess.

Re:Thoughts On Machine Learning Accuracy by MR4D on Saturday July 28, 2018 @09:26AM (#17633047)
And when this tool is pointed at a poor black neighborhood, do you think people will question the results just like they did the members of Congress?

To me, this is the issue. We make all sorts of statements when it’s congress, but when it’s real life it’s not enough to make the news.

After all, how many members of Congress have been shot because a police officer thought their cell phone was a gun?

THAT is what the ACLU is trying to point out here.

Re:Thoughts On Machine Learning Accuracy by ryanmonroe on Friday July 27, 2018 @06:21PM (#17629924)
Taking results of 5% false positives coming from a tool with a 80% confidence level as anything other than "it worked better than expected" is misinterpreting the results.
Re:Thoughts On Machine Learning Accuracy by sgt101 on Saturday July 28, 2018 @04:36AM (#17632248)
How many police forces would (without this debate) have ever done more than use the default settings?

BTW . Why is the default 80% Does it deliver 4/5 TP in the wild? Has anyone done an evaluation? What about vs. the kind of corruptions that are common in the wild - how does it handle odd light, reflections, shadows?

Re:Thoughts On Machine Learning Accuracy by mful on Friday July 27, 2018 @06:46PM (#17630079)
Yes, they did. It's great that they explained their config choices, so that folks with domain knowledge can evaluate reliability of their test, and report back.

For those of us without domain knowledge, however, what the ACLU published is wildly misleading, to the point of being actively disingenuous. What are folks without baseline STEM education, much less AI education, supposed to take from the ACLU article other than "Amazon AI is racist and dangerously ineffective"? Do you think they are opening up the Rekognition docs to evaluate if the default config is appropriate for this test?

(By the way, I wouldn't be surprised if Rekognition is actually racially biased. I also feel a bit icky criticizing the ACLU here — they do great work, and I encourage everyone to donate.)

Re:Thoughts On Machine Learning Accuracy by nerdponx on Friday July 27, 2018 @05:12PM (#17629454)
And yet, if the ACLU never ran their article, we wouldn't be having this conversation right now. I say it was a job well done.
Re:Thoughts On Machine Learning Accuracy by ryanmonroe on Friday July 27, 2018 @05:56PM (#17629752)
The only important part of this conversation is the part that offers a suggestion for the level of confidence you should use when highly accurate matches are important. But we never needed a conversation for that, it's in the documentation (and in the mind of anyone with basic knowledge of statistics, which should include anyone using this statistical tool for anything important).
Re:Thoughts On Machine Learning Accuracy by obastani on Friday July 27, 2018 @04:15PM (#17628961)
But a huge part of the problem is that many users of facial recognition software will not be experts on how to properly use the software. If the ACLU improperly used it, do you really trust law enforcement agencies to do a better job?
Re:Thoughts On Machine Learning Accuracy by l9k on Friday July 27, 2018 @04:21PM (#17629012)
The local cop will not use AWS APIs by himself just like that, connecting it to the surveillance cameras and mugshot databases.

Hopefully, and I think this scandal will help, the engineering departments/contractors for the government will develop software that is appropriately configured for each case.

Re:Thoughts On Machine Learning Accuracy by stochastic_monk on Friday July 27, 2018 @04:38PM (#17629147)
I don’t think a cop or juror is going to look at the confidence score and accurately understand the reasonable doubt that a classification is incorrect.
Re:Thoughts On Machine Learning Accuracy by l9k on Friday July 27, 2018 @05:09PM (#17629432)
The cops won't just consume the API. Software developed for the specific task will.

What I imagine is this: 1. The cop uploads a picture of the suspect taken from camera surveillance at crime scene 2. Software displays the mugshots (and names) of PROBABLE matches from the database with confidence %, color coding according to confidence level, and notice of proper use 3. Cop compares man/woman's picture with what's displayed 4. Investigation continues if necessary

Re:Thoughts On Machine Learning Accuracy by visarga on Saturday July 28, 2018 @02:30AM (#17631887)
Face recognition is not being touted being as accurate as DNA testing or fingerprints, so why pretend it is anything but a fuzzy ranking by similarity? Many forensic techniques generate probabilistic results.
Re:Thoughts On Machine Learning Accuracy by bpchaps on Friday July 27, 2018 @10:39PM (#17631200)
Hah. You clearly haven't worked with any government agencies or their contractors. There will be mistakes, and it won't be just a few. The public will have very little action towards even validating whether they're configured properly. It's gonna be a shit show.
Re:Thoughts On Machine Learning Accuracy by confounded on Friday July 27, 2018 @04:39PM (#17629154)
What’s an appropriate configuration for your neighborhood?
Re:Thoughts On Machine Learning Accuracy by sgt101 on Saturday July 28, 2018 @04:39AM (#17632258)
I would be on Amazon's side if their terms of service clearly stated "do not use this tool for law enforcement". Do the terms of service state that?

If this is used to misidentify someone and then that someone gets shot how will the folks at Amazon feel about that? To me failing to consider that and failing to act appropriately is reckless. It's as reckless as Uber turning off the Lidar and as reckless as Tesla using the road markings as a dominant steering method.

When you have to ask "how many innocent dead will it take?" do we not all think that something might just be going wrong!

Re:Thoughts On Machine Learning Accuracy by TeMPOraL on Saturday July 28, 2018 @07:23AM (#17632680)
> I would be on Amazon's side if their terms of service clearly stated "do not use this tool for law enforcement". Do the terms of service state that?

Why on Earth would they put something like that in their ToS? If anything, the place for that is in documentation ("this product may not be suitable for use in law enforcement"), but then again, since when Amazon does have a clue about the specific requirements of government services, to issue blanket statements like that?

> It's as reckless as Uber turning off the Lidar and as reckless as Tesla using the road markings as a dominant steering method.

Yes, you don't blame the Lidar company for equipping their products with an off switch, nor do you blame whoever designed the road marking detection for Tesla's decision to base their self-driving tech on it.

Re:Thoughts On Machine Learning Accuracy by sgt101 on Saturday July 28, 2018 @09:08AM (#17632990)
I think that this illustrates the challenge of this debate. I believe that your perspective is that Amazon are there to make money, provide a component and then the person using that is responsible for the outcomes. However I think that the person providing the service is responsible for who they sell it to and what for. The analogy that I would draw here is with guns; if you sell a gun to a child you should be accountable for that as a misdeed. Note - that's an analogy to try to convey the point I am making, I'm not comparing the misdeeds.
Re:Thoughts On Machine Learning Accuracy by komali2 on Saturday July 28, 2018 @11:34AM (#17633569)
In your example though, is Smith and Wesson responsible for the store selling the gun to a child? (In this specific case yes, because they lobby against common sense gun control)

For law enforcement, I assume they wouldn't be hitting the Amazon API directly, but rather using software someone sold them which as part of its service uses Amazon's API.

Re:Thoughts On Machine Learning Accuracy by jhowell on Saturday July 28, 2018 @01:01AM (#17631635)
> Shits so polarizing these days it seems there's no room for negotiation, as much as I'd like there to be.

Far worse if you're being polarized. Also they ACLU hasn't released their methodology, are you jumping to conclusions with the ACLU "doesn't understand technology"?

Re:Thoughts On Machine Learning Accuracy by ekianjo on Friday July 27, 2018 @10:11PM (#17631098)
> champion of the people's rights,

The former ACLU, yes, but not the current ACLU: they have repeatedly said they dont want to protect Free Speech anymore.

https://reason.com/archives/2018/07/26/liberty-makes-us-unfr...