Consider that most driving is done feet away from another vehicle. If those cars start being replaced by self-driving cars, then the you are safer (system safety x personal safety). The car might make more dubious decisions than you would yourself, but now it has less opportunities for failure due to others driving erratically.