It would absolutely happen this way.
Like any change, driverless cars would cause a lot of people to lose money and/or their livelihoods. So there will be a significant number of people (and businesses) who fight against the change.
Look at climate change -- the actual metrics don't even matter. There is always a way to skew the display of the numbers to make a pretty graph that will convince enough of the population that there's at LEAST some legitimate controversy over the issue; which is enough to just put the decision off until the controversy is over.
Such a car would be immensely safer than what we have now. Would people willingly ride in one?
I think I'm pretty good at rationally weighing the risks, but just imagining this still gives me the willies.
Imagine if self-driving cars were at a point where they were 10% better (in terms of fatalities caused) than human drivers. You start flooding the roads with them, and fatalities decrease.
Then the news stories start. "Mother of Four Killed by Self-Driving Car." Video of the grieving children. News shows host "experts" who go on at length about the deficits in computer judgment. The family sues, with their lawyer saying that this death would not have happened with a human at the controls. (And he might be right! Even if better than humans, computers will probably get into different accidents, not just fewer.) Restrictive regulations or even bans start being discussed in Congress.
Maybe it wouldn't happen this way, but it could potentially set things back by years. Looking at the technology, pushing for early adoption might save lives. But looking at the politics, pushing too hard could cost lives, by making people unwilling to accept the technology.