Although a driver is not at fault in every accident they are involved in. Therefore it is possible that the good driver would be safer with an increased self-driving environment even if that individual's odds of causing an accident increase as long as the odds of other people causing an accident decrease.

I have nothing but a hunch to back it up, but I have a feeling getting the worst drivers off the road would provide a greater benefit to public safety that would outweigh the damage caused by downgrading the skills of the best drivers.