Bad weather and unexpected obstacles like pedestrians are actually trivial to handle. Remember that an autopilot's "visual" perception isn't limited to the visible-light range like ours is.

Poorly-maintained roads are the problem that would give me nightmares if I worked on these. Lane markers that might have been perfectly usable when the database was compiled may be too worn out or obscured by snow, leaves, or who-knows-what else to be usable later. How can you design a system to fail safely when every input source is independently fallible? And how can we redesign the auto insurance industry to make it possible to try?

>Bad weather and unexpected obstacles like pedestrians are actually trivial to handle.

Google's self-driving cars still have trouble with heavy rain and snow (or at least they did as of last year). http://www.technologyreview.com/news/530276/hidden-obstacles...

Detection of abnormal conditions might be tractable (though I wouldn't call online vision or recognition trivial problems). But taking action is harder. A piece of debris might manifest in a way that causes a self-driving car to skid to a halt or swerve - dealing with abnormal conditions in real time on crowded roads is certainly not a trivial problem, especially given the stakes.

I'd probably lump "obscured by snow, leaves, or who-knows-what else" into the inclement weather category :)

I'd be curious to hear what the insurance industry makes of all of this too, they must be preparing for it in some way or another.