There are behavioural issues as well: people deliberately cutting in on autonomous vehicles because (rather than aggressive tail-gating) the AV will just back safely away, so that dangerous behaviour will become more commonplace. Before we get onto pedestrians just stepping into the road in the serene assurance that the AV will shudder to a halt for them, to the discomfort (possibly injury) of the vehicle's occupants. I can see people doing that kind of thing just for a lark, which would encourage drivers to go for the "aggressive" firmware upgrade to put those jaywalkers back in their place.
Personally I think (a) we'll get these vehicles sooner or later, but (b) they won't be a panacea, and (c) it's going to be an absolute minefield in technical, social and legal areas.
I'm not. Not at all. Sadly.
I'm still feeling surprised by the number of intelligent people who are quite capable of thinking critically, quite uncritically deciding that self-driving cars are ready for the market now. The only thing which comes remotely close is Tesla's recent software upgrade, and Elon Musk was very careful to emphasise that it in no way replaces the driver. Yet still, the hype-train is running so fast that people decided what he really said was "self-driving cars are here now."
A moderately successful prototype from Google or anyone else does not mean the problems are solved. These innovations could still go the way of 90s-era virtual reality goggles, flying cars or the personal jet-pack. Don't get me wrong, it's exciting stuff but there are still significant technical issues, not to mention the legislative and social problems which will inevitably arise.
These links from a Google search for "Google self-driving car problems":
http://gizmodo.com/6-simple-things-googles-self-driving-car-...
http://www.nytimes.com/2015/09/02/technology/personaltech/go...
http://www.technologyreview.com/news/530276/hidden-obstacles...
There's much more, I can't dig out the links now. But don't be one of those guys in 10 or 15 years time being tedious about "Dude, where's my self-driving car?" It might not happen, and it's certainly not here yet.