When the oven will burn you if used incorrectly, and Amazon gives police departments (who don't know how ovens work) the controls over the oven regardless of their provable technical ability and with no oversight, we should definitely throw away the oven.

Maybe yes, maybe no. But realistically, with the hype out there about AI solving all problems, do we think its not going to get used?

Seems like more productive conversation could be had over how to use, than blanket dismissals. The proverbial 400-lbs man in his basement can code this up relatively easily with existing tools, to near state-of-the-art results. So the guy that works for the FBI will too. Let's come up with appropriate, targeted safety mechanisms so the oven doesn't light the whole building on fire.

Not only that you can't uninvent a technology that can run on cheap hardware with open source software and which can be reimplemented from scratch in a week from arxiv papers, but this is a clear case of improper use, where the solution is to set better norms.

Or more precisely, take the flaky oven away from people who misuse it in high stakes situations.

I don’t remember us being asked if we wanted to live in the oven, but here we are anyway.

Good luck getting law enforcement to properly interpret machine learning results. Look at how they still use disproven methods like bite marks to unjustly imprison innocent people.

The product in question is Rekognition, which matches photos. It would be more verifiable by naked human eye than bite marks?

As long as the results are not taken as judgements, but only auxiliary tool, it would not be a concern.

The article indeed argues that any positive match should be vetted by a human.

> The article indeed argues that any positive match should be vetted by a human.

I have said before and I will say again: after the disaster that was the Amazon Fire Phone I have zero trust in any Amazon.com employee to pick up that any time phone to sound the alarm or if they do, I don't expect Amazon to "do the right thing". The reason I mention this is because I don't think this PhD fellow speaks for the the sales "engineers" at Amazon.com. The sales people will say what they have to say to make a sale.

Lets face it: we programmers just work at a company, the business actually runs the show. I have zero confidence that the customer will understand that "any positive match should be vetted by a human". What does vetting mean anyway?

No police has ever stopped me for matching the profile of a suspect, ever. However, I know people who say it happens (or at least used to happen) to them at least twice a year.

Is a blanket ban on law enforcement from using facial recognition possible? Or at least restrict it to federal agencies (a ban on all law enforcement that is not a federal agency)?

That's actually a very good point, sales people will try to exaggerate the capabilities of the technology to the dangerous levels... Well said.

How about "we should not throw away the medical scanner because it can be set wrong and it burned this child"?

Or, don't use the oven on setting 3 because if you do your house will burn down?