Think about it the other way: would you rather a below average driver behind the wheel? or a self driving car?

Everyone likes to think they're better than average (https://en.wikipedia.org/wiki/Illusory_superiority#Driving_a...).

I would rather keep the worst drivers off the road than keep the best.

Then again, bad drivers often make themselves known on the road and you can give them wide berth when recognized (granted, not always possible). Watching for the warning signs of a bad / reckless / distracted drivers quickly became an almost unconscious habit for me.

A machine suddenly hitting a bug is whole new ball game though. I guess over time maybe you could build up a sense of where machines have trouble driving and adjust accordingly.

If you want to keep the worst drivers off the road, it might make sense for the elderly to get self driving cars. Or cars to detect your blood alcohol level and go into self driving mode then. That solved, I believe the average driver would go back up enough that self driving cars are worse.