I hadn’t considered this before but you’re absolutely right. Driving is a continuous operation so it’s almost impossible to pick up without the prior context.

I still think general AI (i.e. AI so advanced that we would consider protecting each instance as we would a human life) is a prerequisite for truly autonomous driving on our complex roadways today. This is the kind of game theory that would make me feel more comfortable sharing the road with these kinds of cars.

I completely agree with the assertion that the grey area in between is where a bunch of people are going to wind up dead.

Maybe not a prerequisite, but certainly necessary to prevent certain type of long-tailed events. For example, I can anticipate by the look of a neighborhood that it's more likely that there would be jaywalking and poor infrastructure, I can feel that the driver riding ahead is drunk or on drugs and keep my distance etc.

It remains to be seen if a automated drivers can make-up for increased risk in such scenarios with good performance in areas they excel, such as reaction time, highly distributive attention, near perfect physical predictions and decisions made on that basis, vehicle-to-vehicle coordination and vehicle to road communication etc.

What is clear though is that any practical system in the near term (non-general AI) will approach driving quite differently than a human driver. Hence the jury is still out of lidar vs camera-only approaches (Waymo vs Tesla).

> I completely agree with the assertion that the grey area in between is where a bunch of people are going to wind up dead.

But will more people end up dead than in the scenario where we had no ‘self-driving’ cars at all?

Humans are much more accident prone than robots in all the research on this I’ve seen.

The thing is, most of the deaths from self driving will be seen as preventable, if only a human was driving. That's going to make it look bad, even if the numbers are less.

If you want to reduce deaths, I think you really want to invest in things like automatic emergency braking, monitoring driver attention, and safely stopping the vehicle if the driver is incapacitated. Having the computer supervise the human means the human is engaged in driving and aware of the circumstances for the most part; a computer supervising a human can act with great speed if the situation warrants, but a human supervising a computer is likely not to react so quickly.

Calculating just deaths / DrivingDistance is easy but overly simplifies the complicated comparison between human- vs self-driving. Comparing normalized deaths is not sufficient to persuade real people to choose self-driving over human-driving. Real humans and society take into account many more factors to accurately compare the two driving modes.

Take a hypothetical scenario where self-driving cars reduce deaths by 10x compared to now, but the deaths caused by self-driving cars are all small children and people with darker skin in human-preventable ways (you can imagine if cameras are used why this could happen). I doubt society would accept this tradeoff.

The safer future is to have humans still drive, but be enhanced by computers and cameras that are installed in the car.

Truth is, we will never accept robots on the roads that are as deadly as human drivers. Especially since they will fail in specific, surprising and hard to comprehend ways, see here.

Robotic drivers will probably need to demonstrate at least an order of magnitude improvement in safety before they can be accepted by the public.

You can probably achieve a safer-than-humans solution without strong/general AI, provided it only operates on a special kind of road (in-highway sensors, no pedestrians, all other vehicles are also self-driving).

Here's the problem. The person was being paid to pay attention to the road. According to the below news article, the person was watching the voice on their mobile phone while the car was driving. Perfectly acceptable if the vehicle has already been rated to be fully autonomous but, in this case, they are still testing the automation and the car detected the bicyclists 5.6 seconds prior to the accident.

Uber should be partially at fault since they deactivated the automatic emergency braking.

Yes, I agree when vehicles become fully autonomous, people will be less attuned to the environment, but in this case, the person was hired specifically to pay attention at all times.

https://news.sky.com/story/uber-safety-driver-charged-over-d...

FYI it was a bicyclist (singular) but they were walking their bike across a crosswalk at 10pm.

Not to victim blame but if they were not wearing lights or a reflective vest I think a human could have made the same tragic error.

As you point out the system actually DID detect them.

This idea needs a little more fleshing out, but is very good.

The Uber vehicle IIRC didn’t abandon control when it saw the pedestrian, it did so (if ever) when it determined that it couldn’t avoid them... So the vehicle had more and better information about the situation.

“Failing to the driver only works when it’s not helpful.“

Not everybody drives the same way -- from how you describe it, to you driving is an all-encompassing activity, requiring presence, planning, attention etc. Don't get me wrong, that's ideal. Unfortunately most people don't drive like that. For many people, it's an incidental activity, a means to an end, done with the bare minimum of resources required to get by. That's why there are so many "accidents" and why you see so much stupidity on the roads.

So, you think SAE J3016 (document that defines this 6-level terminology) is pointless and we could just get away without it? You need to prove your case further.

Your dichotomy may make sense in theory, but in practice where the average driver makes a fatal mistake at rate X, a level-4 autonomous vehicle with a failure rate Y<X (including when human override is required) is an win.

I think your claim is far more theoretical in nature. In a vacuum, sure, fewer deaths is better. In the real world, where people aren't satisfied unless someone can be held accountable in a clear way, we are far better off with humans driving.

The maker and or operator of a self driving car can be held accountable depending on the specifics. IMO, we have this odd social expectation that humans killing people with cars or pollution is basically acceptable and nothing else really gets that pass.

IMO, killing someone with a car or a gun is a direct equivalent.

It's equivalent, yes. When someone recklessly kills someone with a car, it's called vehicular manslaughter and they are punished for it. It's not considered acceptable.

Blurring the lines of accountability is an anti-feature. The killer feature for humans is that we can be held accountable.

That’s assuming it’s prosecuted which is rare. However, just looking at the laws directly you notice the odd exclusion.

In California for example vehicular manslaughter is specifically excluded from involuntary manslaughter and carries a lower penalty. It is also frequently just a misdemeanor.