>In this incident, the driver was using autopilot in a fashion that it should not be; twisting road at high speed. The driver IS at fault.

I disagree on this point. In any system that's supposed to be sophisticated enough to drive the car and but also carries a giant caveat like "except in these common driving situations..." failure is not instantly down to "user error." Such a system should gracefully refuse to take over if it's not running on an interstate or another "simple enough" situation; otherwise, as many have noted, the threshold for "is the driver paying sufficient attention" should be set much, much lower.

That the system is lulling the users into bad decisions is not automatically the fault of said user. Some blame, maybe most of the blame, has to fall on the autopilot implementation. When lives are on the line, design should err on the side of overly-restrictive, not "we trust the user will know when not to use this feature and if they are wrong it is their fault."

It's not lulling anybody into anything! The guy was given multiple warnings by the car that what he was doing was unsafe. The last of which was extremely pertinent to the crash. To quote the article, "As road conditions became increasingly uncertain, the vehicle again alerted the driver to put his hands on the wheel". This is after the initial alerts telling him that he needs to remain vigilant while using this feature. This isn't some burden of knowledge on when to use and when not to use the system, it's plenty capable of knowing when it's not operating at peak performance and lets you know. At that point, I'm willing to shift blame to the driver.

I just want to see "the vehicle again alerted the driver to put his hands on the wheel" followed by "and Autopilot then pulled to the side of the road, turned on the hazard lights, and required the driver to take over."

>and Autopilot then pulled to the side of the road

"It's a winding road going through a canyon, with no shoulder"

So going from a dangerous situation to another dangerous situation.

That's stupid. Autopilot doesn't have the ability to determine if it is safe to pull over. It has a fallback mode; eventually it will come to a complete stop in the middle of the road and turn the hazards on. But since this is also dangerous it makes sense to give the driver time to stop it from happening.

It's stupid to come to a gradual slowdown with hazard lights, but it's not stupid to keep blazing away at speed, with an obviously inattentive (or perhaps completely disabled) driver? I'm confused - even you acknowledge it has that fallback, but it's 'stupid' to enact it after a couple of failed warnings? When should that threshold be crossed, then?

That is exactly what it does; apparently the accident happened before it could complete its "slow down and stop with the hazard lights on" process.

so far we have been lucky the accidents have only impacted the driver. is it going to take the first accident where an autopilot car kills another innocent driver before people see the problem?

When autopilot kills more innocent drivers than other drivers you can point me to a problem. Time will tell if it is better or worse and the track record up until May/June was going pretty well.

I'd rather 10 innocent people die to freak autopilot incidents than 1,000 innocent people die because people in general are terrible at operating a 2,000-3,500lb death machine. Especially because everyone thinks of themselves as a good driver. It's fairly obvious not everyone is a good driver - and that there are more bad drivers than good.

Maybe I only see things that way because there have been four deaths in my family due to unaware drivers which could have easily been avoided by self-driving vehicles. All of them happened during the daytime, at speeds slower than 40mph, and with direct line of sight. Something sensors would have detected and braked to avoid.

Assuming we stick with the same conditions as in this case, I'd call an innocent life taken due to a driver disregarding all warnings from his system negligence. That also fits with what this driver was ticketed with which was reckless endangerment. We don't even get heads up display warnings about not driving into oncoming traffic, but nobody is going to blame a dumb-car for the actions of its driver. I would say that inaction in this hypothetical case would be the driver's crime.

The Tesla warnings remind me of Charlie and the Chocolate Factory, where one of the brats were about to do something disaterous and Gene Wilder would in a very neutral tone protest "no, stop, you shouldn't do that".

From a risk management standpoint, Tesla's design doesn't strike me as sufficient. Warnings without consequences are quickly ignored.

The car crashed. Sounds like a pretty strong consequence to me.

I guess I just don't know how many flashing warnings somebody has to ignore for it to finally be their fault.. At some point, their unwillingness to change their natural inclinations in the face of overwhelming evidence that they need to should have a negative consequence. This isn't like "just click through to ToS". If you buried it in the ToS or something similar, I absolutely agree with your point of view, but when you make it painstakingly clear in unavoidable manners I think we've crossed into territory of user responsibility. If this feature needs to be removed from Tesla cars (and I accept that potential future) I would definitely put that "fault" on the users of Tesla cars rather than Tesla based on the implementation I've read about.

> In this incident, the driver was using autopilot in a fashion that it should not be; twisting road at high speed.

I don't understand why, if the car's GPS co-ordinates are known, the car even allows autopilot on anything but roads known to be within spec.

> The driver IS at fault.

I'm not sure legal liability is that clear cut.

I can't imagine that every road in the world can be determined to be "safe" or "not safe" via GPS coordinates alone.

Then the classification would fail-safe (i.e. no autopilot for you) and that's good[1]. The alternative with the current technology is apparently depending on humans to decide (poorly). This is to minimise deaths caused by engaging autopilot at the wrong time while you gather more data.

[1] If the goal is to stay out of court. I understand the AI drivers are better on average argument.

Certainly the software making the driving decision is capable of 'knowing'

Tesla's "autopilot" system and implementation has serious problems. I think it's becoming clear that they did something wrong. Whether it rises to the standard of negligence I'm not sure, but it probably will if they don't make some changes.

With that said, I agree that ultimately the driver is at fault. It's his car, and he's the one that is operating it on a public road, he's the one not paying attention.

One of the laws of ship and air transport is that the captain is the ultimate authority and is responsible for the operation of the vehicle. There are many automated systems, but that does not relieve the captain of that responsibility. If an airplane flies into a mountain on autopilot, it's the captain's fault because he wasn't properly monitoring the situation.

>>In this incident, the driver was using autopilot in a fashion that it should not be; twisting road at high speed. The driver IS at fault.

It would be disastrous for the car to take control when it shouldn't. That itself is a giant red flag to me.

Its all or nothing when it comes to automation that involves human life.

Sorry, but you don't get off that easy when your design is intended to make life and death decisions. I work on FDA regulated medical devices and we have to do our best to anticipate every silly thing a user can (will) do. In this case, the software must be able to recognize that it is operating out of spec and put a stop to it.