That's impossible. This has been tested in countless studies with train drivers and pilots and it is absolutely impossible for a human to stay alert if an 'almost good enough' computer system is in the drivers seat.
By the time you're needed your situation awareness will be nil.
Even if this was constantly happening, it would give the safety driver some sense of purpose - their job would be constantly figuring out "is this a real thing or not" - and then they wouldn't be bored out of their mind and be watching videos.
I agree with that, and the NTSB should consider adding this to their requirements when approving test programs of this sort.
But stepping back, I think there's a very significant difference in culpability between "safety driver couldn't react in time because they zoned out" and "safety driver was watching a sitcom". In the first case, the driver was trying to do their job, and the nature of the ask made it difficult/impossible. In the second case, the driver was knowingly not doing their job, and was knowingly engaging in unsafe behaviour. We don't have any examples of a fatal accident involving the first kind of error, and this case is an example of the second kind.
> "something is moving closer to the car in a way we're not expecting" would be a great thing to show the safety driver.
Isn't that what you get by looking through the windshield using your eyeballs?
Maybe, but the bigger picture is if you hire people for low wages and give them impossible tasks, you're not paying enough for them to make a good attempt or be a scapegoat. The problem is management.
If they are truly trying to stick to the guns of "well the 'driver' should have been prepared to take over at zero notice", then their (potentially criminal) negligence is in how they present and market their product and not in the software. But it's one or the other.
In some cases in the most literal sense of that.
Yes, if this code was in a Tesla, it would be criminally negligent, because in the driver's seat of a Tesla is a consumer that bought a product called "autopilot".
Uber, and Waymo, and Cruise, are all _testing_ their systems, and that's why they installed safety drivers. There's no marketing here, and there is no customer -- the person in the driver's seat was employed to sit there and pay attention to what's going on, for the explicit reason that the car's computer is NOT yet capable of driving itself.
That's in marketing materials rather then the legal ones. Which is great to make regular people interpret it "look, it can drive itself, I'll just fiddle with my phone" but doesn't put any of the legal responsibility on the manufacturer's shoulders. I'm guessing regulators will eventually have to weigh in on this.
It’s not reasonable for a human to sit around for hours doing absolutely nothing, then suddenly be thrust into an emergency with seconds to respond without warning. Most humans aren’t capable of that level of vigilance. If they aren’t watching a video, they’ll be daydreaming. It’s very unlikely they will be able to recover from any kind of situation requiring an immediate response. At a system design level, by placing a human in that role you have already failed.
Maybe if you take the most qualified humans with the best reaction times, they would have a chance, but the qualifications for that job would be more like those of a fighter pilot. This job is not recruiting our best and brightest, it’s recruiting people who want to sit around doing nothing.
This person’s role in the system isn’t to provide safety - it’s to absorb liability. Ergo, a fall guy.
Counterargument: the safety driver is supposed to be prepared to take control at any time, should the vehicle do something unsafe.
It seems to me to be clearly criminal to use this code to run a fully automated environment (i.e. no safety driver). It's not clear to me what the expectations should be of the code when there is supposed to be an attentive safety driver in the vehicle.
I think the safety driver is going to jail, because they were watching videos instead of watching the road at the time their vehicle killed a pedestrian.