News for hackers. Stuff that matters.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by mannykannot on Saturday August 05, 2017 @08:59AM (#14935479)
Whenever this sort of issue comes up, there are a number of responses pointing out that humans can be fooled too, and I am guessing that the generally unstated implication is that therefore it is not a big deal. Two important differences that this point of view overlooks are that humans are usually able to tell when they are not fully understanding their visual input, and they are able to respond appropriately, which includes acting cautiously and taking actions that will help resolve the uncertainty. Artificial visual systems that have, at best, only a rudimentary understanding of what they are looking at, are in no position to act this way, and can assign high confidence values to what we would regard as ludicrous interpretations of the scene.

These things will be fixed in time. There is nothing to be gained by pretending that they are not problems.

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by DannyBee on Saturday August 05, 2017 @11:17AM (#14936103)
They aren't real problems because nobody smart is trying to directly hook up a road sign classifier to a steering wheel. They are trying to build complex systems where this is just one signal.

If they were trying to directly use this info, this would be the least of the serious issues. For example, there are plenty of stop signs well hidden by trees. Such a car would probably last about a mile in a non freeway setting.

A trivial model of "hey, does this make any sense appearing at this place on a map" defeats this, etc.

Of course this stuff is it is infancy and can do dumb things. But this is just isn't that big a deal.

Humans also act much worse than you imply. Look at the number of people who accelerate into crashes instead of braking, etc. Again, doesn't mean AI should get a pass, and humans are infinitely more complex, but we shouldn't be held up as amazing at this either.

(That's actually the part that worries me. That we aren't good at it)

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by mannykannot on Saturday August 05, 2017 @11:29AM (#14936175)
The fragility of AI decision-making in general absolutely is an issue. Feel free to post a submission when it has been solved to a level that matches or exceeds human capabilities.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by rm999 on Saturday August 05, 2017 @02:17PM (#14937059)
I build AI/ML systems for a living and it's not as big of a deal as you think. Production machine learning systems built by experienced teams will almost always have business logic protections built-in to stop the machine from doing something totally stupid. For example, an auto-bidder may have capped velocities or auto-shutoff mechanisms (those examples of bidders spending billions of dollars in minutes are exceedingly sloppy).

This is not unlike the human body's deeply-evolved reflexes - before the neocortex can even process that a pan is boiling hot, the nervous system bypasses the brain and and tells the hands to drop it.

In the case of driverless cars, DannyBee is absolutely correct: truly driverless cars will not make decisions from a single data point. I have some past experience with unmanned aerial vehicles and those things had insane amounts of redundancy in almost every signal and decision making process; driverless cars are more complex and will come decades later and will probably take this even further.

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by oneshot908 on Saturday August 05, 2017 @03:46PM (#14937521)
As another such practitioner, why oh why would we ignore hard-learned unambiguous signals and wisdom because we jumped on the machine learning bandwagon? We embraced machine learning to detect and uncover the things we don't see, not to replace the things we already know. There's no conflict here whatsoever.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by mannykannot on Saturday August 05, 2017 @04:43PM (#14937804)
> Production machine learning systems built by experienced teams will almost always have business logic protections built-in to stop the machine from doing something totally stupid.

In the case of driving, however, the problem of deciding what behavior to veto is itself a hard problem - perhaps the hardest of all. When the system monitoring the AI is itself AI, the argument becomes circular - which is not to say it cannot be solved, but I think it makes it harder to dismiss the concerns.

[dead]
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by dboreham on Saturday August 05, 2017 @01:52PM (#14936910)
Let's start with Spam : filtering it out with 100% guarantee of no false positives has still not been achieved.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by MR4D on Saturday August 05, 2017 @09:43PM (#14939210)
But also true of humans. In fact, I would guess I that advanced spam filtering engines might be _better_ than the average human. Remember, the Nigerian scammers are still around because someone is dumb enough to still give them money.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by gambler on Saturday August 05, 2017 @02:51PM (#14937246)
> They aren't real problems because nobody smart is trying to directly hook up a road sign classifier to a steering wheel. They are trying to build complex systems where this is just one signal.

What are the guarantees that those higher-level "complex systems" aren't going to have some weird behaviors as well?

The problem isn't just that ANNs misclassify adversarial examples. The problem is that it's counter-intuitive to an average observer and that no one clearly knows why those examples generalize so well. It's a remarkable property that a lot of AI "enthusiasts" try to downplay.

It's one thing to wire up unreliable but simple and fully understood components into a more reliable system. It's an entirely different level of challenge if the "unreliable components" are complex and poorly understood.

> Humans also act much worse than you imply. Look at the number of people who accelerate into crashes instead of braking, etc.

Except we know very well how and how often people make mistakes. For most of those mistakes we have a pretty good idea why they happen (at the high level, I'm not talking about neuroscience). Our roads, our cars and our laws are designed to handle these failures. Also, we have a pretty good model of how other people behave on the road, so we cal react accordingly.

All of this goes out of the window with self-driving cars.

Hell, most people naively assume that if a single self-driving gets into 50% less accidents than an average person than replacing all drivers with self-driving cars will reduce the global accident rate by at least 50%. This assumption doesn't take into account the fact that many accidents are caused by complex interactions between several vehicles and the environment. So introducing many self-driving cars can lead to some nasty emergent behaviors and some mass accidents that just aren't possible with human drivers.

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by samstave on Saturday August 05, 2017 @12:07PM (#14936358)
Hw difficult would a GPS spoofer be, i.e. to transmit the wrong GPS coords via very cheap devices to at least confuse the heck out of any automated car?

Oh look:

http://www.rtl-sdr.com/spoofing-gps-locations-with-low-cost-...

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by DannyBee on Saturday August 05, 2017 @01:03PM (#14936646)
Again, this assumes this is the only input, and things can't detect when some inputs seem completely insane with high probability and raise an error

Seriously. Y'all realize how much in your daily life would pretty much explode and kill people if that wasn't true, right?

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by pjscott on Saturday August 05, 2017 @01:34PM (#14936819)
It's worth mentioning that car GPS navigation explicitly calculates how plausible each GPS reading is, based on a combination of other position signals and basic physics modeling, and discounts anything that sounds too crazy. If it didn't, simple things like tall buildings would be huge problems.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by kevin_thibedeau on Saturday August 05, 2017 @04:55PM (#14937846)
Tall buildings are huge problems for GPS. Multipath reflections will inflate the uncertainty and can put you some distance from your true location.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by pjscott on Saturday August 05, 2017 @10:55PM (#14939434)
Yes, they're a serious problem. What I'm saying is that they'd be a much bigger problem for GPS-based car navigation if there weren't logic in place to combine the GPS info with accelerometers and gyroscopes.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by samstave on Saturday August 05, 2017 @01:17PM (#14936716)
Yeah, but that wasn't my point.. my point is that these attacks will be tried as soon as opportunity>cost-to-attempt
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by kevin_thibedeau on Saturday August 05, 2017 @04:49PM (#14937819)
Tesla did it for their billboard detection. Didn't turn out so well and their "fix" is a horrid hack.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by ajmurmann on Saturday August 05, 2017 @10:27AM (#14935875)
I think another big issue is that humans can tell when a sign is manipulated or damaged to the point where it will be problematic to identify for other humans and it now can be corrected before something goes wrong. If it's only misleading to computers, chances are humans won't fix it till something went wrong.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by microcolonel on Saturday August 05, 2017 @11:39AM (#14936223)
Well, given the example in the article, any human with functioning vision would be able to identify that stop sign. A driverless car with inadequate detection would run through the intersection without stopping.

I'd say that's a pretty big deal.

I think the problem with the driver-less car is that it doesn't have goal-oriented general perception like human beings do. Humans filter the world through goals. The world is interpreted as a set of affordances and threats to your goals, your brain is predisposed with distinguishing the two.

As it stands, this is not how driver-less cars are being designed, they are being designed to specifically recognize the presence and content of street features like signs and other moving bodies. A human driver might see a tall parked car with a big gap in front, and recognize a pedestrian affordance which he should steer clear of in case somebody pops out. The way I see people talk about driver-less cars, this "common sense" type of judgement would have to be deliberately integrated.

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by IanCal on Saturday August 05, 2017 @03:53PM (#14937562)
Is the important thing that humans and machines classify them differently? Or that one or both might be fooled? Because the latter already exists as a problem but doesn't seem to be a major issue. You can already, with physical access to a sign, change what it looks like to fool any observer.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by rsync on Saturday August 05, 2017 @11:03AM (#14936038)
"These things will be fixed in time."

... and the solutions will be written in blood.

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by skybrian on Saturday August 05, 2017 @12:04PM (#14936344)
Yes, as happened with trains, automobiles, and airplanes.

But I suspect that driverless cars will start out pretty safe and improve from there.

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by rsync on Saturday August 05, 2017 @03:00PM (#14937287)
"Yes, as happened with trains, automobiles, and airplanes."

I think there is an interesting distinction that is worth discussing ...

At the advent of train/plane travel, you could opt out of those novel risks by eschewing those modes of travel.

However, you could be driving along in your driver-piloted car, minding your own business and find yourself involved in an accident caused by the software in a driverless car you are sharing the road with.

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by skybrian on Saturday August 05, 2017 @04:05PM (#14937615)
Consider that trains cannot stop quickly or turn and safety consists almost entirely of making sure people get out of the way. And yet, lots of people (even kids) would walk along tracks, etc. And yet there are still urban areas with active train tracks running along surface streets, for example near Jack London Square in Oakland.

Before cars, people routinely walked in the middle of streets. "Jaywalking" wasn't a concept. It was entirely up to early cars to stop in time, and they didn't have good brakes. Fortunately they weren't very fast.

Also, early pilots, besides the obvious risks, were sometimes daredevils who would do things like buzz houses.

I think the main point here is that society's attitude towards risk has changed; we take small risks much more seriously than they used to when earlier forms of transportation were introduced. Accidents involving driverless cars make the national news and manufacturers are being very cautious. With earlier attitudes towards risk, I expect they'd be in widespread use by now.

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by resu_nimda on Saturday August 05, 2017 @03:18PM (#14937383)
I don't think that is an important distinction because that is still very true of driver-piloted vehicles as well. Also people still die in train and plane accidents. You still have to eschew any form of travel to be safe from its risks.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by civilitty on Saturday August 05, 2017 @11:07AM (#14936059)
As opposed to bunnies and in kittens like they are now?
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by megablast on Saturday August 05, 2017 @05:56PM (#14938131)
We are talking about cars, where over a million people die every year. We have already accepted a huge death bill for convenience is ok.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by visarga on Saturday August 05, 2017 @09:19AM (#14935527)
> Artificial visual systems that have, at best, only a rudimentary understanding of what they are looking at, are in no position to act this way, and can assign high confidence values to what we would regard as ludicrous interpretations of the scene.

It's not a good idea to generalize like that. There are many AI systems that use neural nets for perception and then use different models, such as conditional random fields (CRF) to see if entities make sense in context.

Not to mention that these adversarial examples don't work well from all distances and angles. As a car changes position slightly, the adversarial example disappears. It's very hard to create an adversarial example that works from all angles on all networks. There is also a special training regimen that fortifies neural nets against adversarial examples.

The problem of adversarial images is closely related to GANs and has been brought into attention by the same researcher, Ian Goodfellow. So it's not a neglected problem - some of the top minds are on it.

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by dogma1138 on Saturday August 05, 2017 @09:37AM (#14935601)
The adversarial examples don't need to work from all distances and angles.

Many street signs especially in rural areas might have a pretty small visibility window and they will only be visible from a pretty fixed angle until you pass them.

So while you might not be able to develop an adversarial example that works for all signs from all distances and angles it wouldn't be hard to develop one for a specific sign. The angle for that sign is known, the distance in which most cars could resolve the sign is also known, the average speed on the road can also be taken into account which gives you a rough idea of the time span the autonomous driver would have to identify and resolve the sign.

Heck if you know the algorithm it wouldn't be that hard to build an app that basically allows you to take a picture of the sign with your phone from the road, select a car or subset of cars you want to jam and it would generate a likely pattern to apply to the sign to jam those cars.

I should also add that in this case some of the examples that do seem to work look to be of the shape and size of common bumper stickers, this is pretty worrisome since these are not that uncommon on street signs, especially in rural areas where they are more or less at reachable heights.

I also wonder how well do these systems deal with graffiti.

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by ethbro on Saturday August 05, 2017 @11:26AM (#14936156)
Wait until AI puts most truckers out of a job and you suddenly have a substantial number of people in rural America feeling resentful towards fully automated freight trucks... adversarial examples are one thing, but I think the US military's experience in Iraq has taught us that a determined, intelligent, and constantly innovating human adversary is a continuing challenge.

Especially given that the cost of crashing an unmanned truck will be 100% capital and 0% human, I expect we'll start seeing some pretty serious criminal penalties passed once the tech gets there.

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by dogma1138 on Saturday August 05, 2017 @12:23PM (#14936451)
Unless that truck crashes into people :)

I think that scenario isn't as likely as unintentional screwups at the moment.

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by ethbro on Saturday August 05, 2017 @02:19PM (#14937071)
Of course. But there's a lot of road between population dense areas...
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by mannykannot on Saturday August 05, 2017 @09:43AM (#14935632)
> So it's not a neglected problem - some of the top minds are on it.

That's why I believe it will be fixed. It is the "it's not a problem" crowd that I take issue with, and I would be surprised if many of the top researchers held that position, unless they are confident that they are close to a solution.

I think you are under-generalizing here. The risk here is not of a sign being mistaken for a gorilla specifically, or even about signs specifically, but the fragility of image recognition in general. Similarly, because the issue is not (just) about adversarial images, the fragility of adversarial images is not much of a mitigating factor.

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by mrtksn on Saturday August 05, 2017 @11:16AM (#14936096)
At the point when the AI operated cars can handle all kind of situations better than humans, they will be intelligent enough to be classified as human beings.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by Houshalter on Saturday August 05, 2017 @04:34PM (#14937758)
The fact is that these adversarial examples are incredibly rare. 1 in a trillion trillion or more chance that they would occur in natural data. We dont know that human brains aren't vulnerable to something similar. I bet if we could back propagate through the visual cortex, we would find similar things. They can also be mitigated a bit by training on them.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by kthejoker2 on Sunday August 06, 2017 @01:11AM (#14939808)
Optical illusions certainly suggest that our visual processing involves trade-offs.
Re:Slight Street Sign Modifications Can Fool Machine Learnin... by pishpash on Saturday August 05, 2017 @03:29PM (#14937432)
I think you are overstating what "understanding" is. It is well known that birds see more of the EM spectrum than humans. We can make -- and indeed there are -- designs that are only visible to birds but are invisible to humans in the sense that our three-cone visual system conflates certain inputs from the physical world. That stop sign can be designed in a way to look like a stop sign to a bird but make us think it is something else entirely. Are we not understanding the stop sign?

No. It's just a fact of parsimonious signal representation that sometimes you get null spaces with which you can manipulate to maximally separate the performance of two very different receivers.

Re:Slight Street Sign Modifications Can Fool Machine Learnin... by aetherson on Saturday August 05, 2017 @05:08PM (#14937903)
No, that's nothing like analogous. Adversarial examples aren't giving machines information that is invisible to humans.