On that time scale you can expect people to have fallen asleep at the wheel several times. They spent literally months looking at their phones rather than paying attention to the road etc. Because here’s the thing most accidents aren’t fatal, for every death you’re going to see a host of accidents that should have been avoided and tens if not hundreds of thousands of mistakes that could have resulted in a serious accident but didn’t.
The ones killed in accidents involving self driving cars will include people who drove responsibly or even defensively in the past. They will also be among higher income earners for the first few years, while self-driving capabilities are still too expensive to be included in the average car.
Moreover, people like to think they're in the set that wouldn't get themselves killed by their own driving, regardless of how poorly they drive.
In that case you are saving the same lives and not causing any new fatalities, but it's not a super cool tech that will attract lots of investment. So the choice to implement self driving was ultimately putting profits over human lives.
Autonomous driving deaths will be much more scrutinized. In the press, by regulators, by competitors, by the companies involved. There's likely to be a lot more data involved. The scrutiny may ironically increase as it becomes safer, maybe becoming a lot like plane crashes, in fact, where each one is scrutinized by groups of experts and fixes are put in place to prevent that kind of thing from happening.
The safer it is, the rarer and more novel the fatalities are and therefore the more resources will be spent on fixing the safety issues.
That is, if autonomous driving is allowed. If it's not allowed because overall irrational fear of autonomous driving, then that feedback cycle can't happen.
So the paranoia isn't necessarily a bad thing as long as it doesn't cause paralysis.
However, with self-driving vehicles, if you sue the manufacture of the vehicle, they have significantly deeper pockets. This is going to effectively keep us stuck in the place where no matter how good self driving cars become, the responsibility remains with the driver.
Things like that Tesla driving straight into a wall and killing its driver. Or not seeing a van right in front of it because the sun was too bright. Things where a bit of common sense (not something AI excels at) might have avoided them.
On the flip side the lives saved will be based on having super human reaction times / situational awareness and will be things no person would have been able to do.
So maybe there’ll be a battle of public opinion (and PR!) weighing these things against each other.