On an annual basis, you "self drive" a staggeringly large number of cars into accident scenes, road construction, traffic jams, police, and just about everything else that could possibly be on the road.

Start talking about 99.998% or so, OK, now it's worth sitting down with the math. But 98% accuracy at avoiding construction/existing crashes/etc is horrible.