You're thinking about the tail end of the distribution. If self driving cars can solve 90% of the mass of the distribution, i.e. driving on highways and urban streets, it would be a significant improvement in terms of efficiency. Of course there will always be morale questions like the one you proposed. But quite frankly, the return on investment of solving the tail end scenarios like that is really small.