This sort of assistance is an excellent use of current technology, and a way to move forward to true autonomy. What needs to stop here is the two-faced, fatally ambiguous marketing, and if manufacturers cannot act responsibly, regulation will be needed (more than the current regulation for responsible driving, which only kicks in after the damage is done.)

That's exactly the sort of task I'd expect an automated vehicle to perform better on. If they can't beat humans at that they're not ready for the public road.

Of course hand-off could be for liability purposes, like

Car: "I'm sorry Dave, I'm about to plow into a school, this is your problem now"

Dave: [looks up briefly from his game of candy crush before dying in a crash]

The problem is, they can't do this today with the necessary classification quality. Elaine was killed, because someone at Uber wanted these cars to drive so badly. It also is about culture and prediction. If the other participants behave differently to your prediction, it can get hazardous quickly at highway speeds. I can't find a better link [0], but maybe it's good enough to get the point across. The term is "warning dilemma" and means that you only have perfect information at the time of impact. Before, the classification if there will be a problem or not is worse, the earlier you ask. But asking early is necessary for the human to have a chance to react.

[0] https://www.researchgate.net/publication/326568066_Towards_a...

From what I remember, Elaine was mostly killed because she stepped in front of a moving car in the middle of the night.

I’m not convinced that’s the best example, since the AI might have actually done better if it had full control of the car instead of defaulting to ‘please take over now’ mode.

Right. I could have stated it better, what I meant was that the (safety) driver was not paying (enough) attention to the road and was basically driving Level 3 (the car drives and it will beep whenever I need to take over, so just relax and browse the web). It was impossible to react to the short notice and the warning was _not_ given 10 seconds ago. I personally think the system had sensed her as a succession of standing targets and never predicted any velocity vector. Braking for static radar targets is not happening, because of the clutter you get everywhere you would not drive at all. So the misclassification and misprediction was not detectable for the system (in it's world model, everything looked consistent) and it warned too late.

If I remember correctly, the cars built in system for emergency braking in this scenario was disabled. So, any other car of that type would have avoided the accident. However, because the AI apparently knew better, it was disabled and someone died.

AI has the problem that not only does it need to observe the environment and determine what to do, but it has to predict the actions of non-rational actor (humans). We aren’t always predictable. AI isn’t always predictable either. The woman who died may have assumed that a car driven by a person would have stopped.

>That's exactly the sort of task I'd expect an automated vehicle to perform better on

If the AI could talk, it wouldn't be so much "I'm sorry Dave" as "I cannae change the laws of physics!"

An AI can think about the situation for several million instructions worth, if it is running at >GHz speeds, but it can't prevent a crash given a few milliseconds, because inertia. And for the same reason, it's futile to hand off to a human.

Press release: "While the self-driving feature of our car had been engaged earlier in the drive, the driver was in control at the time of the crash."

Humans already kill 3000 people every day while driving

Politics is emotional, and emotions do not always agree with Utilitarian ethics.

Self driving doesn’t just need to be safer, people also need to feel that it is.

Sorta. Self-driving cars cannot a injure or kill someone in any situation where a human wouldn’t have.

I feel this too. I would rather have a greater risk of death but be in control of my fate.

Americans drive more than 3 trillion miles per year.

Who would the car optimize for? Saving the kid or saving the driver from getting rear-ended or swerving into an object?

See kid => hit brakes. This is already way better than a driver who doesn't notice the kid and doesn't hit the brakes. The inside of the car is very well designed to keep the occupant alive in case of collision, the car should prioritise not hitting other people. I hope self-driving technology will at some point be regulated regarding what the car should attempt to do in those situations. With more vehicles having collision avoidance technology, the car behind you won't rear-end you because its AI will react fast enough while also refusing to drive dangerously close to the car in front.

I wouldn't have asked the question if it was that simple :) If you have no other cars around you, are traveling less than 25mph and have plenty of stopping distance, sure, 'see kid => hit brakes' should work to save everyone in that circumstance.

If your assumption is that you won't get rear-ended because everyone will be using automated collision avoidance technology... Well, we're at least 15 years away from that.

As someone who bikes and is used to seeing every other person on their phone while driving, fully self-driving vehicles can't come soon enough. But we need to have real discussions about who the self-driving vehicles will save in regular circumstances. I'm skeptical about them optimizing for, or even having the ability to optimize for saving people outside of the vehicle over protecting the occupants.

Ok now bonus Problem. The car has to swerve and either hit a kid or an old lady, which do you hit and why?