And a scenario we can easily imagine is that a buggy update goes out to the whole fleet overnight that starts killing people all over the place.
The common case of accidents being on par with manual human driving goes out the window until the software is rolled back and for 12 hours, 24 hours, however long, we get a number of deaths that far outpaces what humans are capable of. The "worst case" would never apply to a manual/human population as a whole, at once.
Well, one of the issues is that someone more or less has to be held accountable. And that someone pretty much has to be the manufacturer. No one is going to hand over full control to a vehicle and accept the responsibility if that vehicle commits vehicular manslaughter because "software isn't perfect."
It's actually an interesting legal situation because, other than maybe drug side effects, there aren't a whole lot of consumer products which, properly used and maintained, sometimes randomly kill people and we're OK with that because sometimes stuff just happens.
Who's responsible if you get pulled over for going 75 in a 65 mph zone?
Also at the very least an self-driving car should reach the level of a good driver, having self-driving cars cause as many deaths as drunk or inattentive drivers do nowadays isn't defensible. Especially since there's usually no explanation and nobody to hold accountable.