It's already the case that pedestrian/car collisions are usually written off as the pedestrian's fault until proven otherwise--the Uber self-driving car that killed a pedestrian is a great example of that in action.
It all has to go wrong at once. Mistakes are common but most mistakes aren't fatal.
And by choosing to drive a car, you are dramatically raising the odds that any mistake you make is fatal (not necessarily to you).
While my personal experience on the matter is thankfully limited, I strongly suspect that the vast majority of vehicular-pedestrian accidents are mostly (or entirely) the fault of the driver, where the pedestrian started an action where it was safe to do so and the driver rapidly caused it to become unsafe before the pedestrian could react.
In general a car that doesn't see a pedestrian isn't a major accident unless the pedestrian also doesn't see the car. If not for that there would be a whole lot more fatalities with human drivers as well. A self-driving car obviously shouldn't do that, but neither should a human, and they do. Which makes it less hard for the computer to do it at least as well.
You can also expect the computers to improve over time by learning from each others' mistakes, which human drivers generally don't.