How does a self driving car deal with a blizzard with white-out conditions or a complete loss of traction due to ice? Yes, humans get into a lot of accidents in these conditions but as a percentage of the time the rate is still remarkably low.
If you want to eliminate large classes of those adverse situations then the best approach we have right now is to switch to rail transport. Public transit is truly the way to go!
The eternal pessimist within me looks at how difficult it is to get anything with infrastructure accomplished, both at a reasonable pace and reasonable cost, in the US and can't help but expect this sort of thing to not play out as wonderfully as that sounds.
We are nowhere near a future where cars exist in hermetically sealed tubes, completely walled off from the rest of existence.
Excommunicating humans from all public space and drawing a moat of monopoly owned robo taxis around every block and every destination isn't a utopia, it's a complete nightmare.
In short, what took 25 minutes door to door on any given day before ended up taking ~two hours on the way to work. And for reasons I can no longer remember, getting back home ended up taking twenty minutes longer.
In other words, a total commute time of ~50 minutes (max) was turned into a ~4.5 hour commute due to having to rely on public transportation.
I also remember the specific reason why I went and bought a new car that same weekend (fortunately this happened on a Friday): it was the realization, upon arriving at home (after said ~2.5 hour commute), that I had forgotten my apartment keys at work.
It's pretty neat, I've moved to a European city recently myself and I'm slightly horrified at the prospect of moving back.
In your hypothetical future, where no humans manually drive anymore, it opens up the possibility of completely redesigning our vehicle infrastructure without having to worry about the training and compliance of a mixed human/AI population on the road.
My optimistic hope is that with fully automated vehicles, mass transit wins out on sheer efficiency in order to negate the wasteful deadheading inherent in personal vehicles.