Maybe not a prerequisite, but certainly necessary to prevent certain type of long-tailed events. For example, I can anticipate by the look of a neighborhood that it's more likely that there would be jaywalking and poor infrastructure, I can feel that the driver riding ahead is drunk or on drugs and keep my distance etc.

It remains to be seen if a automated drivers can make-up for increased risk in such scenarios with good performance in areas they excel, such as reaction time, highly distributive attention, near perfect physical predictions and decisions made on that basis, vehicle-to-vehicle coordination and vehicle to road communication etc.

What is clear though is that any practical system in the near term (non-general AI) will approach driving quite differently than a human driver. Hence the jury is still out of lidar vs camera-only approaches (Waymo vs Tesla).

> I completely agree with the assertion that the grey area in between is where a bunch of people are going to wind up dead.

But will more people end up dead than in the scenario where we had no ‘self-driving’ cars at all?

Humans are much more accident prone than robots in all the research on this I’ve seen.

The thing is, most of the deaths from self driving will be seen as preventable, if only a human was driving. That's going to make it look bad, even if the numbers are less.

If you want to reduce deaths, I think you really want to invest in things like automatic emergency braking, monitoring driver attention, and safely stopping the vehicle if the driver is incapacitated. Having the computer supervise the human means the human is engaged in driving and aware of the circumstances for the most part; a computer supervising a human can act with great speed if the situation warrants, but a human supervising a computer is likely not to react so quickly.

Calculating just deaths / DrivingDistance is easy but overly simplifies the complicated comparison between human- vs self-driving. Comparing normalized deaths is not sufficient to persuade real people to choose self-driving over human-driving. Real humans and society take into account many more factors to accurately compare the two driving modes.

Take a hypothetical scenario where self-driving cars reduce deaths by 10x compared to now, but the deaths caused by self-driving cars are all small children and people with darker skin in human-preventable ways (you can imagine if cameras are used why this could happen). I doubt society would accept this tradeoff.

The safer future is to have humans still drive, but be enhanced by computers and cameras that are installed in the car.

Truth is, we will never accept robots on the roads that are as deadly as human drivers. Especially since they will fail in specific, surprising and hard to comprehend ways, see here.

Robotic drivers will probably need to demonstrate at least an order of magnitude improvement in safety before they can be accepted by the public.

You can probably achieve a safer-than-humans solution without strong/general AI, provided it only operates on a special kind of road (in-highway sensors, no pedestrians, all other vehicles are also self-driving).