To "work", as self-driving car has to be able to deal with all contingencies, such as unexpected heavy traffic, pedestrians, kids, drunk college students, sudden rain, sudden fog, constructions, police redirecting traffic, etc.

Oh, and it also shouldn't kill anyone more than once in 100 million miles or so. Even if some trips turned out to be slight less simple than predicted.