I bet I could cause a significant fraction of human vision systems to get in a crash with a well placed sticker. I'd replace "<- One Way" with "Detour ->".

I'm pretty sure this would fail to kill people on almost every place you could try it. And if it works somewhere, it's because there are other problems with the road that should be fixed.

Human driving is full of redundancies, and there is a clear hierarchy of information. People will not rush into a road full of cars going on the other way, it doesn't matter what the signs say.

If your automated driving system doesn't have those same features, it's not ready for use.

> People will not rush into a road full of cars going on the other way, it doesn't matter what the signs say.

And people would not drive into a river passing through multiple barriers, just because their GPS says so.

https://theweek.com/articles/464674/8-drivers-who-blindly-fo...

https://indianexpress.com/article/trending/bizarre/driver-in...

A few people taking completely stupid decisions is always going to happen. There's also drunk people or elderly people with poor vision who end up in the wrong direction on the highway. Even if it happens every day on a global scale, it's an incredibly small fraction of drivers. A targeted attack fooling current-technology IA could lead hundreds of cars in the wrong direction at the same time in a single place. It's way worse than what you could imagine tricking humans to do. Putting such brittle systems in charge on the road is irresponsible.

Hundreds you say? Think again, it's happened. (Well, 100)

https://www.google.com/amp/s/www.cbc.ca/amp/1.5192656

That's a fun trivia, but that's not really relevant to the discussion here: those people took no risk for their lives when following these instructions. Making hundreds of people go in the wrong place is easy, but the difference between a bad IA and a human driver is that the majority of drivers realize when they're doing something dangerous and stop.

This common sense of danger is what IAs must have before we can trust them on the streets.

For instance in the Uber fatal crash: the IA correctly labelled the cyclist as a cyclist several times then revised its judgment, then definetely labelled it as a cyclist but “oops too late”. A human would be like “wait, I think I saw a cyclist, slow down and figure out what this really was”. Obviously with the current computer vision technology you cannot do that, otherwise your car would be afraid of everything and refuse to even start, because there's way too much mislabelling happening at some point or another.

I just meant it as an example where a large group of humans will follow what a computer tells them even when it is pretty clearly wrong. It's not hard to conceive of possibilities where the danger wouldn't be obvious until after person blindly follows the instructions.

> A human would be like “wait, I think I saw a cyclist, slow down and figure out what this really was”.

This explains why there are only 5.25 million car accidents a year in the US.

Most of them being benign collisions though. “Only” 34k of them are fatal. For like 270M vehicle in circulation.

I don't know how many automated vehicle Uber has, but with one killed already the ratio is a few orders of magnitude worse for their automated vehicle than for the average driver in the US. (probably even way worse than the average drunk driver actually)

> People will not rush into a road full of cars going on the other way, it doesn't matter what the signs say.

You might want to watch the one-way roads in big cities. It happens a lot more often than you assume.

It also is (usually) self-correcting: oncoming traffic will honk, stop, or move around. The offender will (usually) realize their mistake and try to correct.

Sometimes, though, that's not enough. Searching "killed in wrong way one way" on DDG (or assumably Google) yields many (!) news stories.

Been there, done that (except the "killed" part). It was in a heavy fog. I was doing well to find a street at all, and it turned out to be one way the wrong way (the only such street in town). I figured it out when I saw wall-to-wall headlights coming at me out of the fog, and made a fast move for the curb...

So, yeah. People react. Which brings up the question: How well do self-driving AIs respond to a wrong-way driver? How well do self-driving AIs recover when they are the wrong-way driver, and they suddenly have enough data to realize that?

I'd suggest a slight correction here: You don't need the qualifier of 'big cities'. I live in a big town/small city it happens all the time here too. And when I lived in a small town that had a one way drive around the square, it happened a lot there too.

It's so frequent in fact that the barflies at one local place (that has a beer garden from which you can see a one way road) turned it into a drinking game - wrong way car == take a shot.

You put too much faith in humans. Things like stop sign removal have caused deaths in the past. https://www.nytimes.com/1997/06/21/us/3-are-sentenced-to-15-...

Humans are not so bad as drivers. Your example is an event from over 2 decades ago and was deemed newsworthy. Humans drive in all kinds of conditions but death rate is about 1 per 100 million miles driven. A search reveals crashes to be on the order of hundreds of collisions per 100 million miles driven. Age, country, intoxication level, road design and laws, road and environmental conditions also play a major role such that accident rates for someone aged 30+ in a Northern European country are going to be a lot less than teenagers in a country where road laws are merely friendly suggestions (and considering the chaos of driving in those countries, the rates are actually surprisingly low).

I will go further and say that almost everytime there is an accident the driver is somehow impaired. Lack of sleep, drugs, illness (old age included, mental disease), poor judgement (young age included, emotional distress)

Humans are surprisingly good at driving under normal conditions.

Some are. Some are not. Last week, I was nearly in two accidents on maybe a 1 mile trip to the store from my house. Both times were people pulling out of traffic, ignoring right of way. I prevented the accidents that would have resulted from these two separate idiots. I have also been in over 20 accidents in my 25 years of driving, the vast majority of those having been rear-ended and none were my fault.

In my experience, I've not been in an accident with a teen, nor someone elderly, though I know people that have (both causing and being involved). Neither have I been in an accident with someone that I could tell was impaired by drugs or alcohol. I don't know for sure any of them involved a phone for that matter. Weather was only a factor in one accident (pouring rain, low visibility).

I have nothing to suggest that any of my accidents were caused by anything other than inattentiveness, even the one time weather played a minor role. I also see a lot of dangerous behavior every time I drive: people running lights and stop signs, completely ignoring yield signs (seriously, they must be invisible to everyone else), failing to yield right of way, failing to signal turns and lane changes (my favorite is turning the signal on after moving into the turn lane), lots of phone usage (for everything except making a call, from maps to texting to watching videos!).

> I have also been in over 20 accidents in my 25 years of driving, the vast majority of those having been rear-ended and none were my fault.

Do you just drive a lot or do you brake too late / too hard? Because an accident rate that high is rather unusual.

No, I don't drive that much. Most of my driving is in bumper to bumper traffic on 3-4 lane roads, though and people do tail gate a lot. None of the accidents have been because I've braked too hard. Of the most recent accidents, two were due to people not checking their blind spots while changing lanes. One, I was waiting at a stop light and the car behind rolled into me. And the fourth, a neighbor backed into me. Most of the accidents I've been in are after having been stopped at lights for nearly the entire duration before the accident occurs. The worst, I got hit by a guy speeding in a 45 mph zone while stopped at a light. He totaled his car. I've also had several hit and runs while I wasn't even in the car.

As I stated in my post, most of them are due to people being inattentive and not following the basics like keeping a reasonable distance checking blind spots. Others I suspect phone use, but can't prove it.

Yeah, the “I'm like 5 sigmas deep in the probability distribution but I bear no responsibility” sounds a bit suspicious.

> or do you brake too late / too hard

You mean live in a place where drivers tailgate?

> I have also been in over 20 accidents in my 25 years of driving, the vast majority of those having been rear-ended and none were my fault.

That's way too many to take your word for it. Where there's smoke, there's fire. Well, maybe not... but you sure as shit should suspect a fire.

> a country where road laws are merely friendly suggestions (and considering the chaos of driving in those countries, the rates are actually surprisingly low)

I wonder how well self driving algorithms would compare if tested in such a "hostile" environment? Perhaps we shouldn't allow them on more "friendly" roads until they can consistently surpass human performance under such adverse conditions.

the claim is not that automated driving systems are ready for use, the claim is that if you do things in order to compromise a system that has a good chance of killing people and then does kill people that should be illegal, which of course it already is.

Yeah. "Voluntary manslaughter" and "malicious mischief" are already things you can prosecute for.

You could spray handful of nails in the road and I think there is a big chance it would cause an accident. Or you could just dig up a hole using tools available in most homes. Agreed, it's not that easy, but not hard as well.

Yeah i think if you start digging in the middle of a busy junction, people will have some questions for you

You would think people would have questions for bike thieves using angle grinders but nope. Just throw on a high-vis yellow jacket.

These attempts to imply a broad equivalence between current machine vision and human capabilities do not hold up under a modicum of scrutiny.

Humans have well-developed models of how things should be, can detect when things seem wrong, and come up with ways to address the apparent anomaly (including taking steps to investigate and evaluate the situation.)

Humans do not always use these capabilities well, but they have them, while similar capabilities are at best rudimentary and fragile in current AI. The premise of this article is that these capabilities will not come easy.

This has always confused me as well. What would be the reason why some adversary would choose to craft an adversarial example and deploy it in the real world versus the much easier solution to just remove / obscure the sign?

Depending on how big or small it needs to be, potentially for subtlety? Especially on current roads that are shared by humans and self-driving systems, a human observer will immediately notice that something is terribly wrong with a replaced sign.

But... around here at least, signs have stickers or graffiti on them often enough. Like adding the name of a politician under a stop sign: "Stop [Harper]". An appropriately made adversarial example won't stick out visually the same way that a wholesale sign swap will.

Because NeurIPS doesn't publish papers on stop sign removal yet :P

Warfare comes to mind, as weapons gain increasingly powerful ai functions and become autonomous

There are multiple reasons for signs to have different shapes, sizes, and colors, and this is one of them.

An orange diamond "detour" sign isn't easily confused for a smaller rectangle "one way" sign.

Additionally, there should always be two large "do not enter" plus two large red "wrong way" signs that are visible to a driver from in the intersection before turning.

Something as simple as tape or other coverings on an existing sign should never result in any confusion as to right-of-way for a driver paying attention.

Some people key off the shape enough that they wouldn't follow a wrongly-shaped detour sign, so you wouldn't fool everyone, but you'd absolutely fool a lot of people. I expect I'd be one of them.

I think in almost all cases that would not cause a crash. The drivers would see the oncoming traffic and stop rather than crash.

that assumes you can see the threat, if instead it led to an unprotected crossing at high speed, then you have a very different situation

But that is not a vision issue. That is providing people with incorrect information.

The specific trick doesn't really matter; the point is that it's possible to maliciously create a situation that makes human pilots act dangerously. We accept that the possibility can't be made nil, and we have post facto rules to deal with it. The same principle applies to traps for machines.

Nope, it 0.1% of humans crash but 100% of teslas crash that's not 'the same'

So would it be better/fine if Tesla randomized the CV ML models to some extent? I actually feel the answer is probably "yes", surprisingly...

Perhaps a mirror/reflective sticker that blinds drivers near a sharp curve?

I see this kind of argument with blockchain bros as well, and it drives me nuts.

If I write crappy paint program and all I can claim is, "It's no worse than the time/effort of drawing by hand," what exactly have I achieved in your opinion?

And if the posts on HN wrt blockchain and ML constantly feature these "no-worse-than-what-we-are-replacing" arguments while posts about, say, paint programs don't, what does that say about the buzz around blockchain and ML?

Edit: clarification

I bet you can't. Humans are anti-fragile and can compensate with other knowledge.

Turn a temporary road sign for 30 speed-limit into a 80 speed-limit with some black-tape (I have already seen it done when people were angry to be fined by speed-detector for a few excess km/h, (or just for the lulz) ). It probably won't fool humans, but it's an edge case that a self-driving car may ignore.