There are behavioural issues as well: people deliberately cutting in on autonomous vehicles because (rather than aggressive tail-gating) the AV will just back safely away, so that dangerous behaviour will become more commonplace. Before we get onto pedestrians just stepping into the road in the serene assurance that the AV will shudder to a halt for them, to the discomfort (possibly injury) of the vehicle's occupants. I can see people doing that kind of thing just for a lark, which would encourage drivers to go for the "aggressive" firmware upgrade to put those jaywalkers back in their place.
Personally I think (a) we'll get these vehicles sooner or later, but (b) they won't be a panacea, and (c) it's going to be an absolute minefield in technical, social and legal areas.