> Letting a computer drive my car is high-risk: If there is a malfunction at 70mph, I could die!

Google can't even make GPS mapping work perfectly - it's apparent if you use it for a day. Not to mention that AI vision systems are susceptible to adversarial attacks, which means they can be hacked by external images. This paper just came out yesterday: "Breaking 7/8 of the ICLR 2018 adversarial example defenses" (https://arxiv.org/abs/1802.00420). The situation is dire.

I think they are afraid that drawing a couple of lines with a marker on a street sign might make it invisible, or turn it into another sign. They can't defend against such attacks because the neural net doesn't actually understand the world, it just observes patterns. Understanding requires causal reasoning.

>Not to mention that AI vision systems are susceptible to adversarial attacks ... The situation is dire.

That's like saying a human driver can be blinded with a bright light so the situation is dire.

New world, old rules. You'll still go to jail for manslaughter (or whatever the charge would be).

Fwiw maps used for autonomous vehicles are higher fidelity than consumer maps. Furthermore the cars are generally equipped with IMUs so that they don't get thrown off by urban canyons, clouds, etc. And more error correction (particle filters, visual cues + map data) is used to pinpoint your location on a map.

Adversarial attacks are concerning for any AI model, but autonomous vehicles aren't all AI. Lots of rules and heuristics and pre-knowledge are utilized. In the end if something is uncertain it generally amounts to "slam on the brakes" which is more a ux problem than a safety problem.

There is no « better quality » map out there (unless maybe military and restricted).

Mapping the world is hard because it is complex and evolve way faster than a Google Car army can patrol it for a reasonable cost. More often than not OpenStreetMap which is a community project is more up to date than commercial map, but it’s not precise enough. Beside nobody would bet there life on something that is Wikipedia-like editable by ill-intentioned trolls.

This is precisely why all autonomous car rely heavily on computer vision and why adversial attacks should be carefully considered.

Uncertainty will indeed « slam on brake » but adversial attack don’t aim to generate uncertainty the aim to generate false certitudes.

Nope. There are higher fidelity maps. You should read about robotics.

It's true these maps don't exist for the whole world yet but it's also why all sdc companies are launching in metros. They're building them.

Vision is good at recognition but bad at localization. This is the real problem. In robotics a fundamental problem is SLAM. You should read about it.

Could you explicit acronyms? They seem very business specific but I would gladly learn about theses projects as I often use cartographic products for my work.

UX problem unless in traffic; getting rammed by the next car is not my idea of safety (not to mention that slamming the brakes for no apparent reason is a textbook example of reckless driving in many places). Oh well, let's just not drive unless the street is empty, I guess ;o)

Well map software on a phone isn't written to specs that account for liability for human life for one. Apples and oranges.

It should be. They're used as inputs to humans making decisions at highway speeds. Now obviously we allow this because we assume humans are intelligent enough and have enough of a model of the real world in their head that they should be able to cope with these issues but I see no reason why an AI driving a car should cross a lower bar. I fully expect that AI to take mapping errors into account and to survive them.

On the flip side, very few people die while mowing the lawn or unloading the dishwasher. If you mow the lawn poorly, you'll be fine. If you unload the dishwasher while drunk, there's no risk of killing a family.

I know enough people that drive terribly to see the obvious benefits to autonomous driving. It doesn't need to be perfect to have a net-positive effect on road safety, it just needs to be better than a human on average. And based on the evidence, it seems they're doing a damn good job of achieving that goal.

I agree, but wanted to point out that you are making the implicit assumption that all drivers are going to be replaced by autonomous vehicles. Otherwise comparing to the average driver may be biased as there most likely are correlations in some way between driver safety and likelihood of acquiring autonomous vehicles (desirability / affordability).

It's not making the assumption that all drivers will be replaced. It's making the assumption that the average skill level of the drivers that are replaced will be lower than the skill level of autonomous cars.

I have a robot vacuum, but: it doesn't do corners. It doesn't do steps. It doesn't get along with cats. It's orders of magnitude less effective than a human vacuum pilot. Yet this is also a problem that's orders of magnitude easier to solve, yes?

Are you willing to pay for more?

Robovacuums seem to have hit a sweet spot - they do just enough on their own to justify their costs by saving you time and effort. Solving the corner cases you mentioned is possible, but there is not a strong enough economic incentive for that.

It's a winner-takes-all market, so there are absolutely strong economic incentives to build the best robovacuum control software.

Why is it a winner-takes-all market?

In fairness, cars move in far fewer ways than a vacuum. Vacuums need to make sharp turns and get into tight spaces on unmarked, uneven surfaces. Cars move forward and slightly to each side for almost all of their movement. Cars also drive on mostly flat surfaces that are well marked and mapped.

You haven't seen an italian town.

Cars make sharp turns and get into tight spaces on unmarked, uneven surfaces, with pedestrians walking everywhere around the car.

That's what a self-driving car needs to support. All road rules out of the window, tiny roads you canbarely navigate, and due to a festival everything is redirected around and all your maps are useless, you're told to drive a short bit through a park because there's a parade on the road, the only way to tell where you can and can't drive being a nonstandard sign and a police officer telling you.

This is a real life use case of driving, as given in many European towns. For additional difficulty, do a small Weihnachtsmarkt in a rural place, where you'll have the same, but with so much snow you can barely see anything.

How well could you, human, vacuum a house if all you were given was a Roomba robot's chassis controlled by you via joystick, and maybe a camera mounted in the robot?

There is a physical challenge arising from the form factor, such that your intelligence won't help. Those robots don't do stairs and tight places very well simply because the hardware is that way.

In driving, the robot has the same controls as the human. It can turn the wheels, brake, apply gas. That basic playing field is level.

It's all about how well you make the AI see and understand, and translate that into turning the wheel, changing forward/reverse, and hitting gas/brakes.

Robot vacuums put very few lives at risk and operate in a manifestly simple environment. They literally navigate by heading off in random directions until the crash into something, then turn. Self-driving cars are an entirely different domain.

(I'm referring to devices/products such as the Roomba.)

Already true, of every system in your car. There are arguable a huge number of things that can go wrong in a car already.

All of those things have a good chunk of a century worth of development (as well as trillions of dollars in R&D) put into making them more robust, building redundancy, etc. I think the parent comment has a good point in that this automated driving thing is a very enormous change with a lot of risk behind it that doesn't appear to be being well managed.

Yes, but there was a time for all of those things in which they did not have a century of development. Deploying them is how you get that century of development. The risk of this stuff is really quite small and local. If the technology has faults, they'll become apparent relatively quickly and harm a relatively small number of people. I see no reason to be overly cautious here. It's not a massive correlated risk, like say, global warming.

We put the new tires and brakes on the the fastest vehicles we could make, then had a professional run them around a track with a dozen other lunatics. Anything that broke was fixed forever, and relatively few lives were lost. The number that died was low because there were fewer participants. The critical technology was well-tested before it was sent to the production line.

That's how we got our century of development. Testing, verifying, refinement. It wasn't by pushing out new stuff to a batch of users and seeing if everything worked out allright.

The number of recalls issued weekly by the car industry gives lie to that claim. No, we just live with the risk. Because we love our cars so much.

Just recently it was discovered a series of air bags manufactured in a small window of a couple of days, would instead of protecting the occupants, would emit high-speed shrapnel into their chest and face killing them. Quite an oversight.

That recall is actually far larger than a few airbags made over a couple days (it's more like 65-70 million airbags) but this is exactly my point. The operating principle of an SRS system is pretty straightforward. Computer detects a crash, computer fires the airbags. It's actually a separate module that's only got the one job. Despite its relative simplicity there are also recalls where it sometimes fails to fire the airbag [1], or sometimes fires it for no reason at all! [2]

If the automakers can't catch problems in a simple system like this, just imagine what sort of new and exciting failures we'll see in an inadequately-tested car autopilot! These things need to be very thoroughly tested before we allow lots of them on the road, and that needs to happen before they omit the steering wheel.

[1] http://www.automotive-fleet.com/channel/safety-accident-mana...

[2] https://www.autoblog.com/2015/10/30/fca-recalls-894k-total-v...

Don’t you think that the number of recalls would be even higher without motor racing being used as a proving ground?

If your airbag fires you’re already in an accident and might die anyway. You’ve already accepted that risk. There are probably better recall examples but I’m not sure it’d disprove the underlying point though.

But isn't that what they are doing right now? The only difference in terms of your argument seems to be that it's not other race car drivers and spectators at risk but other drivers and pedestrians. Then again they are not racing, but testing autonomous vehicles with drivers as a backup.

The only difference.

Other than that, Mrs. Lincoln, how was the play?

What's the reasonable test surface for a new type of brake or tire or electrical system? These are things that you can reasonably well do some fundamental engineering analysis on and say "that seems like a good design" or not. It's also something that you can test in a way that gives you a high confidence that your tests are sufficient to provide high confidence in the design. And even then there are problems that slip through.

Switching from a human driver to software automated driving is a much bigger change in automobiles than has ever come before. It's also one with a much larger reasonable test surface, by orders of magnitude. Systemic risk? How about meltdown and spectre vulnerabilities? The failure mode of most new hardware in cars requires either edge cases or corner cases to result in the worst case scenario of a crash at speed. And yet that's a very common failure mode in autonomous driving. You can't realistically have a car that can autonomously drive itself 90% or even 99% of the time, it must be nearly exactly 100% of the time to be worthwhile and safe.

Automatic Lawn mowers are a thing.

https://www.husqvarna.com/au/products/robotic-lawn-mowers/au...

I've had an automatic lawn mower for years, not sure what your point is there.

Then why don’t we make driving completely illegal? If you make an error at 70 mph you die and you will kill other innocent people. And it happens with a scarily high frequency today. Self driving cars are supposed to bring drastically down the number of car deaths, and I think this is a much more important goal than having an automatic lawn mower or a robot that unloads your dishwasher (seriously??).