Because we have the driving records of all self-driving vehicles. Based on the evidence, their accidents per 100,000 miles driven is far superior to the average driver. The law of large numbers tells us this is highly unlikely to be the result of luck.
We do? Because one of Uber's apparent reasons for testing in Arizona rather than California (FWIW, Waymo has since moved from AZ to CA) is that there are no requirements to disclose driving/performance records, as there is in California [0].
The most we know at this point is that Uber has claimed to have driven 2+ million miles [1], which means it has a fatality rate of 50x the rate of human drivers (~1.18 per 100M miles). Yes, it's not fair to extrapolate like that, but this is just to point out that you need to cite evidence for "[self-driving vehicle] accidents per 100,000 miles driven is far superior to the average driver"
[0] https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/auton...
[1] https://www.forbes.com/sites/bizcarson/2017/12/22/ubers-self...
I'd like to believe this technology will save 20000 lives per year. But today is a time for humility and reflection.