> Counterargument: the safety driver is supposed to be prepared to take control at any time, should the vehicle do something unsafe.

That's impossible. This has been tested in countless studies with train drivers and pilots and it is absolutely impossible for a human to stay alert if an 'almost good enough' computer system is in the drivers seat.

By the time you're needed your situation awareness will be nil.