Even if autopilot was faulty in some way, if more people were living than dying - does it matter?
Similarly, 9/11 would be a footnote compared to sketchy pain killer drug trials. The black lives matter movement would focus more on sentencing disparities and for-profit prison conditions than police shootings, etc, etc.
The reason it matters here is that it's presumably easy for Tesla to fix any bugs that are killing people.
That is, Autopilot makes certain kinds of accidents more likely — those in which the human driver is not monitoring the vehicle carefully enough, Autopilot misses something critical, and an accident results. It is possible, though not necessarily the case, that these accidents tend to be more serious than accidents that Autopilot prevents, such as lane-changing bumps or other less-fatal accidents.
To be clear, I'm not saying that evidence has been presented that the type of accidents prevented are less important than the type facilitated. I'm just pointing out that not all accidents are equal, and that there may be correlation/causation that makes the overall calculus more complicated.
Now, someone makes a robot that has 100% success rate, but every 100 operations, it has a glitch in the sofware, where it stabs and kills the patient instead. It has been calculated that on average, using the robot only kills 1-2 out of 100 people going in for surgery, compared to the old method of 80 people out of 100. The robot is clearly 80 times better than manual procedure, so why not keep using it?
The answer seems to be - because robot killing people is not an unavoidable outcome. It can be fixed. It's a software problem, a mistake that costs someone their lives. Similarily, an auto-driving Tesla may be killing less people on average than manual drivers, but that doesn't mean that any software problems that lead to it don't matter.
Step 2: Correct the bug, so the robot stops killing patients.
The Tesla autopilot may be in a similar situation. Even if there is a number of fatality inducing bugs, disabling it would be even worse. That said, 40% crash reduction is not enough. I want to see 75%, 90%, and more.
Don't even speak of the bug. Just state what matters: this stuff is better than what we had before, and it can (and will) be even better. This may prevent talks about killer robots.
The biggest difference between an autopilot accident and a traditional accident is fault. Most accidents are driver error. Even if it drives accidents, overall, down (which it will), changing the responsibility for safe travel from mostly the individual to mostly the manufacturer has huge legal repercussions. Not to mention many people's discomfort with no longer being in control of their own safety, even if study after study shows that they shouldn't be.
The therac arguably saved way more people than it killed. However... it's one of the most infamous software bugs in history.
It's the same story you came up with above... except it's real, and it got pulled.
They did however improve since then.
Apart from that, more people living than dying could also imply that certain groups of people, maybe kids or bikers, could be at a higher risk of dying while the majority has a much lower risk, resulting in a net positive. So the acceptance of self-driving systems would also depend on the distribution of the risk.
If you let your friend drive you somewhere in your car because you think they're a better driver, is it their fault if they crash?
The reason could be anything - you're drunk and they're not, you're in really congested unfamiliar area near where they live, whatever. If you think they're statistically going to be a better driver it does not absolve them of liability. Someone is always to blame.
Autopilot will reduced the number of fatalities in the same way that going to the hospital when you are sick will reduce the number of fatalities. However some people that use Autopilot will die when they otherwise would not have if they hadn't used Autopilot, just like how some people going to the hospital will die from infection when they otherwise would not have if they hadn't gone to the hospital.
But it doesn't mean that people dying from faults "don't matter". Every HAI-caused death is a tragedy, and they certainly matter.
The issue is much less black-and-white than the GP's simplistic view.
There's no requirement that driver assist software magically become self-driving software in that situation. Put another way: it's nice if the software can avoid a problem, but not required.
The main remaining concern is that drivers not be confused about what they are required to do; there's discussion about that in this document.
This is probably why Musk pulled radar and vision processing in-house. Combining the data from both sensors is more effective, but more difficult, than combining the results from processing both sensors separately.
It took a combination of problems to cause that crash. The police lieutenant who had informed the railroad of the parade in previous years had retired, and his replacement didn't do it. The police marshalling the parade let it go through red lights. They were unaware that the traffic light near the railroad crossing was tied in to the crossing gates and signals. That's done to clear traffic from the tracks when a train is approaching before the gates go down. So ignoring the traffic signal took away 10 seconds of warning time. The driver thought the police had taken care of safety issues and was looking backwards at the trailer he was pulling, not sideways along the track. People at the parade were using air horns which sounded like a train horn, so the driver didn't notice the real train horn. That's what an NTSB investigation digs up. Those are worth reading to see how to analyze a failure.
[1] https://www.ntsb.gov/investigations/AccidentReports/Pages/HW...
[2] https://www.ntsb.gov/investigations/AccidentReports/Pages/HA...