If you have years of experience driving a car, maybe it'll work. For some time. But what if your last manual intervention was five years ago? Or twenty years ago? The rarer handoff happens, the more stress it puts on the manual operator, and the more likely things go wrong.
It's much better to have a safe default reaction (stop the car, shut down the reactor, ...) that kicks in when normal, automatic operation can't continue.
If you haven't already, I highly recommend listening to the Econ Talk podcast for much more depth from someone who has been thinking about these kind of problems for years (I can't really do it justice in a HN comment).
Which isn't really any different than any other sort of major mechanical failure.
We're already okay with the possibility that a human might drop dead in the middle of an interstate and lose control and more humans might be injured or killed as a result. As long as it is not more likely that a self-driving car will drop dead, I don't see why it's any different.
(And this isn't even getting into things like the possibility of catastrophic failure being lower and the options for recovery being much higher. You can run two robots with redundant hardware; you can't run two humans quite as easily. A human who loses vision will lose their concentration; a robot that loses vision can use its last known positions of nearby cars to attempt to pull over immediately in an orderly fashion.)
It's the same principle. You try to do the safest thing you can, but yes, at some point a car will fail and be stuck in the middle lane of an interstate because the autopilot fails, just as can happen because of mechanical failure.
1) It hands control to an out-of-car human driver, similar to how military drones and driverless trains are operated now.
2) It turns on the hazard lights and pulls to the side of the road. Another vehicle is dispatched to pick you up.
To me, it makes sense to make use of the brain power provided by the occupant of the vehicle. For example, if one of the main vision sensors in the car breaks, why not let the human occupant drive the car manually?
[0] http://www.amazon.co.uk/Our-Robots-Ourselves-Robotics-Autono...
[1] http://www.econtalk.org/archives/2015/11/david_mindell_o.htm...