The devil is truly in the details. You're covering 99.5% of all driving situations with 99.9% reliability? Well, then the remaining bit is going to be pretty devilishly hard! What do you do about cars that are stuck stationary on the the side of a bit of shoulder-less road? Here's what a human being does: They "sense" that something is up on the road ahead because the traffic patterns are different. They observe what the drivers ahead are doing, then start exercising caution concerning what's going on at the side of the road, maybe even to the point of hugging the other side of the lane out of caution.
Are "driverless" cars doing this now? No. Are you going to have to develop human-equivalent AI to do that? No. But is taking things like this into account going to take a lot? Damn straight!
http://www.theatlantic.com/technology/archive/2014/05/all-th...
The key to Google's success has been that these cars aren't forced to process an entire scene from scratch. Instead, their teams travel and map each road that the car will travel. And these are not any old maps. They are not even the rich, road-logic-filled maps of consumer-grade Google Maps.
They're probably best thought of as ultra-precise digitizations of the physical world, all the way down to tiny details like the position and height of every single curb. A normal digital map would show a road intersection; these maps would have a precision measured in inches.
Edit: Figures from 2015 indicate human interventions to prevent an accident were about once per 74,000 miles or maybe slightly worse than humans but not that much. http://www.roboticstrends.com/article/google_self_driving_ca...