Problem is that self-driving company would get blamed for accidents for all of its cars. So imagine if you had 100,000 cars going around and you got sued for accident on every one of them. This aggregation of blame is core issue which otherwise gets distributed among millions of human drivers. This invariably requires that self-driving car must be many many magnitudes better than regular humans.

Even if companies figure out way to not get sued, its only matter of time until some very serious tragedy happens such as pregnant woman wearing same dress as color of sky crossing the road getting killed or car running in to school kids. Then there are obvious malicious usage such as modifying car sensors to fool self-driving system and purposely run in to people (cars as weapon scenario). One such thing and it could likely be the trigger for large public outcry, heavy regulations and finally game over for self-driving cars.

I think it might be more desirable to approach self-driving cars in more evolutionary fashion. We can start with self-driving only in less than 25 miles/hr scenarios such as heavy traffic OR sunny days on highways. Then we can start equiping our road networks with dedicated self-driving lanes, supportive beacons on roads and so on. Then gradually move towards make all lanes self-driving.