In planes, when the autopilot disconnects because of some problem there's usually plenty of warning (including a sound like https://www.youtube.com/watch?v=8XxEFFX586k ) and the pilot(s) have some time, seconds or more, to assess the situation and take control. Despite how fast planes fly, the rules of separation and general vastness of airspace make it much easier to smoothly transition control. It may not sound like much time, but consider that in a self-driving car, the latency required to avoid a collision could be in the range of milliseconds. By the time the driver realises (s)he has to take control, it's too late.
Let's also think about the diminishing skills that we'll experience once automation takes over the majority of our driving. Imagine taking 3-4 weeks off from driving and then being asked to take the wheel under a difficult circumstance.
This is also called "living in New York"
Imagine what would be involved in doing that.
Some analysts think that trucking would be an easy foot in the door for this technology. Freeways present fewer challenges in driving. There could even be dedicated lanes for autonomous trucks. The problem is political. A lot of peoples' livelihood depends on trucking. Those people won't go quietly into the night.
Uber and Lyft want to use this technology for commercial passenger service. They've already disrupted taxis. Dropping drivers would be less of a political hurdle for them. I imagine Uber setting up driverless cars in pool routes like buses. Ride services are also the best hope for completely driver-free tech because no one is going to want to assume driving liability as a passenger.
Completely driverless is where this technology needs to go and I think that taxi/ride-service replacement is the lever. Once that works you'll see people buying cars that don't let you drive.