Imagine if self-driving cars were at a point where they were 10% better (in terms of fatalities caused) than human drivers. You start flooding the roads with them, and fatalities decrease.
Then the news stories start. "Mother of Four Killed by Self-Driving Car." Video of the grieving children. News shows host "experts" who go on at length about the deficits in computer judgment. The family sues, with their lawyer saying that this death would not have happened with a human at the controls. (And he might be right! Even if better than humans, computers will probably get into different accidents, not just fewer.) Restrictive regulations or even bans start being discussed in Congress.
Maybe it wouldn't happen this way, but it could potentially set things back by years. Looking at the technology, pushing for early adoption might save lives. But looking at the politics, pushing too hard could cost lives, by making people unwilling to accept the technology.
It would absolutely happen this way.
Like any change, driverless cars would cause a lot of people to lose money and/or their livelihoods. So there will be a significant number of people (and businesses) who fight against the change.
Look at climate change -- the actual metrics don't even matter. There is always a way to skew the display of the numbers to make a pretty graph that will convince enough of the population that there's at LEAST some legitimate controversy over the issue; which is enough to just put the decision off until the controversy is over.
Such a car would be immensely safer than what we have now. Would people willingly ride in one?
I think I'm pretty good at rationally weighing the risks, but just imagining this still gives me the willies.
I see not only Google, but a number of auto manufacturers, doing a lot of work. The Google testing, which is on a relatively small number of precisely mapped roads in good weather, has had a quite a few incidents that required a human to intervene (which won't be a realistic option once normal drivers turn over control). So from my perspective there's quite a bit of work going on but a lot more experience and work is needed before this is something that can be sold to consumers even for limited use cases.
(What we are already getting in high-end vehicles is assistive driving systems which should reduce accidents as they become more widespread--at least so long as they're not so effective as to increase issues associated with inattentiveness.)