While these modifications are fascinating for being something that humans can't readily notice, the underlying attack is something you can also trivially perform against a human driver by covering, removing, or altering a stop sign such that it no longer looks like a stop sign, and it will be much more effective than trying to trick self-driving cars.

In the case of a human not seeing a stop sign, some drivers will remember a stop sign used to be at that location, but a self-driving car is less likely to forget and might (in a world with networked cars, which I find slightly scary for different reasons than this) be able to "remember" that other cars had previously seen a stop sign at that location.

I don't think it would be "much more effective" than trying to trick self-driving cars. Humans are (still, much) better at reasoning their way through ambiguous sensory input based on their abstract understanding of the world than AIs are.

While an AI might specifically already know that a given intersection is supposed to be controlled, there are large numbers of driving situations in which memory (or maps) aren't very useful in dealing with.

When it comes to self-driving cars, probably the way around attacks on signs or other static road features is just to fail safe even in situations in which you're mis-directed by road features (that is, you drive cautiously enough that even if you don't understand the road situation, you can find a way to stop without hurting anyone). But that's not a general AI solution, and may not even be a general self-driving car solution.

Except no amount of specially crafted noise will trick a human. If it looks like an octagon and is red then a human will interpret it as a stop sign. That's because I'd hope as a human you have a better conceptual understanding of what a stop sign is and what its intent is. Whereas none of the current crop of AI can claim any kind of understanding or even yield any kind of explanatory model.

Deep neural nets should really be called "really complicated composition of differentiable functions" and all the training algorithms should really be called "really complicated function root finding" because that's all these things are at the end of the day, a rats nest of functions with a billion knobs.

> That's because I'd hope as a human you have a better conceptual understanding of what a stop sign is and what its intent is.

That's not the reason at all.

It's because if you add exactly the right type of specially crafted noise to an image of a stop sign such that, to a human, it looks like a panda--then who is going to be the arbiter that decides "No, it's still actually really an image of a stop sign and not a panda". The machine?

The real reason is the subjective nature of the perception of reality. With the advent of AI, all sorts of schools of philosophy that used to seem like useless navel-gazing are going to find some real important practical applications real soon. This is metaphysics and ontology. On a similar note (not in TFA), in the "friendly AI" debate, we're going to be looking real hard at the foundations of the philosophy of ethics (where does it really come from?), not just the teleological vs deontological (etc) debates we use to reason about law, politics, governance and appeasement of our "justice" instincts.

> Except no amount of specially crafted noise will trick a human

Or quite possibly we just don't know how to construct the noise yet :P

Well we do. Conspiracy theories are noise but not in the same category as pixel noise. When neural nets start falling for conspiracy theories then I'll start to worry about the coming singularity.

Might as well traffic signs were intentionally chosen as something human are good in classification of without any regard to machine learning algorithms (nonexistent at that time). Yet there are definitely tasks where specially crafted noise can fool humans. Think optical illusions or this car prototype camouflage: https://arstechnica.com/cars/2017/02/the-new-mclaren-720s-wi...

> covering, removing, or altering a stop sign such that it no longer looks like a stop sign

Isn't that illegal in most jurisdictions?

I would rather expect the same laws that keep you from defacing or removing a stop sign would apply to modifying one with noise to make it invisible to self-driving cars.

Although, you could argue that an attacker who is targeting self-driving cars could modify the sign in a way that is unnoticeable to humans, but greatly affects how the self-driving car interprets the sign.

Eg. look at the panda images at the start of the linked post. The third image looks like a panda to humans (and most humans probably wouldn't be able to see the difference between the first and third images), but it greatly affects the machine learning interpretation.

By modifying the sign in this way, if it is possible, it would be much harder to detect and enforce against than it would be if the attacker was targeting human drivers.

I get what you are getting at here. The human isn't the interesting bit. Humans have pretty good bullshit detectors built in already.

An image that triggers a false positive within the AI won't be noticeable to the average human, however good at detecting bullshit as they are. It probably looks like some annoying abstract pattern at the worst.

To the car, however, it could represent anything that made it react adversely upon observing it. Seems like figuring out how to find those images might pose a challenge.

Already exists - "The triangles are known variously as 3-D, virtual or just plain fake speed humps."

https://nudges.wordpress.com/2008/07/14/another-visual-trick...

The comment there suggests calling them “bump l’œil”, which is genius.

Humans will quickly learn that these things are fake and just drive over them at full speed, though. It might stop people new to the area, but not the majority of people who live there.

I think that's reasonable but perhaps not that important. In a school zone, the neighborhood people presumably send their kids there/have friends kids there, and are likely to slow down anyways. Even if you don't, it could be a useful reminder given that it's a school zone to really take it slow. It's not a perfect solution but with how much cheaper it is perhaps still viable.

I wonder if such "patterns" could find use in clothing or as bumper stickers. I can envision this sort of thing taking off as a counter-culture, or anti-technology social weapon. It certainly wouldn't be hard to produce and iterate on.

I imagine it would be hard to enforce, let alone legislate against subtle visual cues that trigger machine vision signals.

Interesting times lie ahead...

About 10 years ago, when I learned about the EURion Constellation[1], I made a T-shirt with the pattern on it. It sort-of worked - a photo taken at the right distance would frequently cause a copy machine to refuse to copy it. It was twitchy, though.

At the time, I was thinking of posing for my DMV photo with it on, because I thought it was interesting and kinda funny. Failed to do so and never resumed the experiment.

[1] https://en.wikipedia.org/wiki/EURion_constellation

People have been using a similar idea to counter facial recognition.

https://cvdazzle.com/

More recently, CMU made glasses that can make you show up as someone else. http://qz.com/823820/carnegie-mellon-made-a-special-pair-of-...

The AI can be trained to become robust to Adversarial examples. Not only that, there is work going on right now that aims to recognize all Adversarial examples. Early results are promising.

I fear for the future if the fascists control the neural networks.

Did you read the article?

> We find that both adversarial training and defensive distillation accidentally perform a kind of gradient masking. Neither algorithm was explicitly designed to perform gradient masking, but gradient masking is apparently a defense that machine learning algorithms can invent relatively easily when they are trained to defend themselves and not given specific instructions about how to do so. If we transfer adversarial examples from one model to a second model that was trained with either adversarial training or defensive distillation, the attack often succeeds, even when a direct attack on the second model would fail. This suggests that both training techniques do more to flatten out the model and remove the gradient than to make sure it classifies more points correctly.

Ah, haven't kept up with Adversarial examples research lately,sorry.

or water transfer printed "makeup"

Everyone can have custom-tailored hoodies that cause the human recognition systems to identify them as Luke Cage.

Ever seen a video of some kids pulling the "invisible rope prank"?

The AI doesn't need to be perfect, just better than humans.

But the problem is that AI can fail in unexpected ways.

Humans are much worse than AI when it comes to self-driving tasks like image/pattern recognition, sure. But they are inefficient in expected ways where measures can be taken like having larger and brighter signs, stronger rules, etc. But what happens when you don't know for sure when it can fail and when it won't?

It's been a few years since people are seeing the magic of machine learning but something like this was just discovered. Are you sure if someone goes in front of a Tesla with a picture like the parent comment quoted, it wouldn't crash and cause harm to the people inside?

So, it just needs to be (significantly) better than the average human. I think it's even better because each time an unexpected crash happens all cars can learn. We can't even teach people not to drive drunk.

Is it that scary to not know how and when it can crash? You don't think about being blindsided at each intersection...

Good unit testing includes maliciously constructed inputs.

Attackers can also shine bright laser pointers at aircrafts. Thankfully we have cops to stop this kind of vandalism effectively.

Do current self-driving systems use vision as ground truth about the presence of things like stop signs? I would hope that the cameras would be used to augment and perhaps feed back into a street map with this information, and that cars would come to a stop at ambiguous intersections.

Potentially they could also just watch for cars on visible cross-streets; even in otherwise ideal conditions, it would be good if your autonomous car didn't keep going when it has the right of way if it can see a car about to barrel through a stoplight from the side.

This brings up a point I've thought about quite a bit with autonomous vehicles. I studied transportation planning in college and one of the tenants is that human drivers are on a spectrum between aggressive and conservative about finding efficiencies in high traffic situations. The canonical example is that if a lane next to you is going significantly faster, an aggressive driver will perform a (relatively) unsafe maneuver to cut off drivers in the faster lane and change lanes.

IIRC, significant research has been performed to identify whether the resulting lane change is net positive (frees space in the slower lane) or net negative (causes cut off car to slow down).

If autonomous vehicles prioritize safety over efficiency by stopping if there's an ambiguous intersection, will that have a net negative effect on traffic while all these self-driving cars slow down for each other.

Couldn't this kind of problem be mitigated by some form of coordination between vehicles? I think a fully automated fleet would have an overall net positive effect, due to, for example, improving the reaction time to traffic lights (or eliminating them altogether).

s/tenants/tenets/g :)

You definitely can't expect map data to be reliable enough to be the only source of truth for something as important as stop signs.

Yeah, but these can definitely be used as complementary sources - with proper integration, they'd match almost always, but all mismatches seem like you should assume that the sign is there; e.g. if a sign is in map but not in DB, then it's likely the sign is obscured by something but still should be followed; and if there is a sign in vision but not in DB, then it's likely to be part of some temporary road works. The scenarios where the sign truly "isn't" there are possible, but comparably much more rare.

The same sort of tactic can be done with humans:

https://www.youtube.com/watch?v=G_pAcIjqcuY

Nothing (and likely on-one) is foolproof.