Which is the status quo for both cars and a bunch of other industries (pharma, tabaco, alcohol, fast food, the list goes on). Nothing is ever perfect (though many things don't have "death" as a potential failure case) but you are only looking at the downside and ignoring the lives that would be saved.
I don't think many families think "Well our Bobby died in a car accident that would have been preventable by technology but even if he had access to it thousands of other people would still die even while using that tech so we are ok with him dying".
The key difference here is that alcohol and tobacco do not autonomously kill people. We can feel safe blaming deaths caused by them on the individuals who purchased and used those products. Same with automotive deaths. As soon as the blame goes from diffuse individuals to a single locus, the torts will come out in full-force. If a self-driving car kills 10% as many people per vehicle-mile as the average individual driver, then the company making it will get sued out of existence.
(Or, if you will, substitute any other concrete existing model of car for the one I named.)
It's a whole different ballgame to accept that computers are going to choose to kill thousands of people per year, and it is not much comfort to know that it's slightly fewer thousands of people. I think this is especially true when the response to such deaths is not "we will investigate thoroughly and ensure this never happens again".
This is not a logically consistent argument, but humans are not logically consistent beings.
While it may be true that perfect is the enemy of good, there is also such a thing as setting the bar too low.