The other thing to bear in mind: I would guess that the vast majority of accidents are caused by people not paying attention -- distracted by a phone, person, great view, or even just because they're looking the wrong direction. Cars don't do any of those things. They will always be paying attention, looking in all directions at once.

So if they eliminate all of the accidents that are caused that way, but increase the number of garden-gnome related accidents...wouldn't that be worthwhile?

Another, less flippant way to say it: yes, we can imagine scenarios in which the number of accidents will increase. But what is the relative frequency of those types of accidents compared to run-of-the-mill rear-end, red-light run, lane change due to inattentiveness?

If given a choice between changing from what you are used to which has drawbacks, to something new that has drawbacks, even if those drawbacks are markedly milder, almost everyone will choose to stick with where they already are. Consciously choosing a position which has drawbacks that you know about makes you feel personally responsible for those when they occur. That is infinitely worse than suffering the drawbacks of the "default" you stay with which you can give yourself a moral pass on because it wasn't a situation you chose to begin with.

There's also the issue of what a self-driving car could do... and what they WILL do. Much technology COULD be amazing... if companies did not function in the ways that they actually do. Because of that, technology often fails to meet any of the promises made. Development is supposed to be cheap and fast, and safety and security get left at the door. No company is held liable for such things, and it would be expensive and bothersome to fix, so they're not about to bother.

And even if the number of accidents increases: Does that increase cause greater monetary loss than the considerable cost savings of not paying drivers.

There's no ethical question here. You either see that it's a gnome, or you see that it's a child.

Humans may or may not be less situationally aware overall, but we have the gnome question down cold. Cars don't. They see "obstacle in front of bumper radar."

The OPs point is well-taken because it illustrates a technical risk that few are discussing: the "last 1%" of self-driving software may be equivalent to solving AGI.

I do think the "last 1%" may be a very hard problem to solve, but the majority of crashes are simply people not seeing things or making bad decisions. "Recognition Error" and "Decision Error" are the two most frequent categories of accidents which basically boil down to not seeing obstacles while performing a maneuver (changing lanes and not seeing a vehicle in that lane, turning and failing to see an oncoming vehicle) and driving too fast respectively. ( see https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/...)

I don't see why the "last 1%" problem needs to be solved when we have hundreds of thousands of accidents every year caused by people not paying attention, not reacting in a timely fashion, and driving unsafely; all problems that computers are likely to excel at.

Because a half-baked product is dangerous (re: handover to human) and unsatisfying. You really want the version with no steering wheel. I’ll take the intermediate one, but let’s work toward the good one.

>the "last 1%" of self-driving software may be equivalent to solving AGI.

Exactly right. You need a semantic world model, reasoning and prediction. AGI bottlenecks self-driving, assistants, text understanding…

I think we are overestimating amount of intelligence we humans need while driving...

The edge cases are unclear: construction zone, heavy equipment, school kids, another car poised to enter the road.

They are probabilistically modeled already. You keep tracking other objects and compute various possibilities for their movement and depending on the past and present behavior pick the most probable one. If car encounters lower-than-threshold probability, it can always slow down or stop, like what non-reckless humans would do. Even end-to-end deep learning can recognize construction zones and learn to drive around traffic cones.

The edge cases are endless (wheelchair woman chasing a chicken), and without a semantic world model the car will have no idea what to do. Low-level object avoidance architecture isn't enough.

Actually that case would be detected as two objects on a collision course and car would stop/slow down. Mind that sensors scan surroundings with high frequency and some of them can see "beyond the walls" like radar. Ultrasonic sensors can also detect the state of road surface (dry, wet, icy etc.).

Yes, but when can the car proceed, and in which direction? Helps to know what a wheelchair, a woman and a chicken are and how they behave. Helps to be able to speak with them.

Superhuman sensors will be great, but you really can't dodge AGI for true full self-driving.

It's probably super easy for the car to stop and open the window for a person to talk ;-)

True, you could have the passenger making "shoo" noises :D

Are humans any better at flying? Doesn't seem to impact how they feel about getting into an airplane.