A few people taking completely stupid decisions is always going to happen. There's also drunk people or elderly people with poor vision who end up in the wrong direction on the highway. Even if it happens every day on a global scale, it's an incredibly small fraction of drivers. A targeted attack fooling current-technology IA could lead hundreds of cars in the wrong direction at the same time in a single place. It's way worse than what you could imagine tricking humans to do. Putting such brittle systems in charge on the road is irresponsible.

Hundreds you say? Think again, it's happened. (Well, 100)

https://www.google.com/amp/s/www.cbc.ca/amp/1.5192656

That's a fun trivia, but that's not really relevant to the discussion here: those people took no risk for their lives when following these instructions. Making hundreds of people go in the wrong place is easy, but the difference between a bad IA and a human driver is that the majority of drivers realize when they're doing something dangerous and stop.

This common sense of danger is what IAs must have before we can trust them on the streets.

For instance in the Uber fatal crash: the IA correctly labelled the cyclist as a cyclist several times then revised its judgment, then definetely labelled it as a cyclist but “oops too late”. A human would be like “wait, I think I saw a cyclist, slow down and figure out what this really was”. Obviously with the current computer vision technology you cannot do that, otherwise your car would be afraid of everything and refuse to even start, because there's way too much mislabelling happening at some point or another.

I just meant it as an example where a large group of humans will follow what a computer tells them even when it is pretty clearly wrong. It's not hard to conceive of possibilities where the danger wouldn't be obvious until after person blindly follows the instructions.

> A human would be like “wait, I think I saw a cyclist, slow down and figure out what this really was”.

This explains why there are only 5.25 million car accidents a year in the US.

Most of them being benign collisions though. “Only” 34k of them are fatal. For like 270M vehicle in circulation.

I don't know how many automated vehicle Uber has, but with one killed already the ratio is a few orders of magnitude worse for their automated vehicle than for the average driver in the US. (probably even way worse than the average drunk driver actually)