> Imho, that's a grave, serious decision, but not one to which the morally absolute is "zero, always."

I think that what we've learned throughout human history is that no one should be trusted and given the power to make such decisions. It has invariably led to disasters. The very idea that someone can decide to put someone else's life in danger for potential future advances gives too many perverse incentives - chief of which, rationalization that "more people will be saved" even as the experiment crashes and burns.

When it's for the greater good, or by the winners, we conveniently omit it from history.

When it's for the greater evil, or by the losers, we publicize it as a cautionary tale of moral hubris.

Example 1: Vaccines

This is an interesting example because the winners and losers seem to be proportionately spread across almost all of society (except anti-Vaxxers who benefit from herd immunity but don't sacrifice with the minute risk of vaccines). See also: https://www.hrsa.gov/vaccine-compensation/index.html

Example 2: War on Crime / Safety

Another example would be the war on crime. Liberties and lives are sacrificed in the name of safety for others. Unfortunately, the liberty/lives sacrificed fall disproportionately into one group, and those protected fall disproportionately into a different group group. If you look at the 2nd order effects (impoverishment, single-parent households, effects on education and future) lots more lives are lost, again in a subset of society.

This is clearly possible in principle. After all, fully autonomous vehicles can have much more capable sensor systems than any human driver, can process the input from those sensors faster, and can then apply any necessary adjustments to the vehicle's behaviour almost instantaneously, all without ever getting tired or distracted. No human driver will ever match that potential.

The problem I have with this whole argument, and thus the reason I'm not sure the moral/ethical arguments stand up either, is that we don't know how to reliably write excellent software yet.

The entire argument for autonomous driving rests on the premise that we will at some point in the usefully near future reach a situation where the cars drive themselves much better than we humans collectively do. But there is, to the best of my knowledge, no evidence so far that we as a society can actually create the necessary software yet.

Until we can, any autonomous vehicle control system will be at risk of life-threatening bugs. In particular, it will be at risk of widespread failures of the same type that could cause much more harm than any single human driver's failure could, whether caused by defects in the software alone or by a malicious actor exploiting a security vulnerability. That is a huge risk that is getting far too little attention so far IMHO.

The first is whether we expect the autonomous vehicles to become safer than the population of human drivers they would be replacing under normal circumstances. This is a population measure, and it's about overall safety and averages, and given enough time and data it seems a realistic outcome.

The second is the risk of a single catastrophic failure, which is a risk that doesn't generally exist with today's human-controlled vehicles. It is no exaggeration to say that the kinds of autonomous vehicle technologies we are conceiving here have the potential, if they go wrong, to cause more harm than any WMD ever deployed in war. And not only the theoretical potential, but given the current state of our software development abilities, a non-trivial chance of it actually happening in practice.

Until we've dealt with the latter problem, which I don't believe we currently know how to solve, any arguments based on the long-term superiority of autonomous vehicles over the capabilities of any human driver is going to be fundamentally flawed.

This is what it really comes down to.

Imagine if we invented the car today in our current cultural climate. I don’t think it would be possible for humanity to reap it’s advantages.

Back when it was invented people actually had to drive the things around on public roads that had a bunch of pedestrians walking haphazardly accross them continuously.

I was an autopilot skeptic, and it’s entirely for this reason. I didn’t think it would be possible for anyone to actually do what needed to be done to get this technology across the line. That is to say, give it to customers before it is safe.

If Tesla is actually able to keep doing this without it getting killed politically, then I think it actually will happen, and that gives me some hope for humanity more generally.

That, I don’t see ever being true. What would you get from that that you couldn’t get from “large amount of real world human driving data, using systems in production”?

Or to put it another way, every time the driver's inputs diverge from the automated system's inputs, the test run is invalidated. You're no longer getting results of what the automated system did.