It never got accepted for publication because the reviewers didn't think it was significant enough, but I wrote a paper presenting an alternative way to analyse autonomous vehicle reliability data (you'll have to take my word for it, but the reviewers didn't quibble about the results presented, just whether it was important):
https://arxiv.org/pdf/1812.08901.pdf
The major technical contribution was demonstrating that a class of mathematical models called software reliability growth models are a reasonable way to estimate the "anomalous behaviour" rates of an AV system as bugs are found and fixed.
My second point was that using fatal accidents as a way to measure safety is not necessary; you can use more common events such as non-fatal accidents, or in the case of AV systems in development, even the number of occasions the safety driver overrode the automatic control.
> In this study, all disengagement events listed in the company reports were included in the analysis. It is likely that in some cases, disengagements occurred as a precaution by the safety driver, and no adverse event would have resulted if the AV continued in autonomous mode.
Take a look at [1]. It says that:
> Waymo has developed a robust process to collect, analyze and evaluate disengages for this report. We set disengagement thresholds conservatively for our public road testing. The vast majority of disengagements are not related to safety. Our test drivers routinely transition into and out of autonomous mode many times throughout the day, and the self-driving vehicle’s computer hands over control to the driver in many situations that do not involve a failure of the autonomous technology and do not require an immediate takeover of control by the driver.
> To help evaluate the safety significance of disengagements, Waymo employs a powerful simulator program. In Waymo’s simulation, our team can “replay” each incident and predict the behavior of our self-driving car if the driver had not taken control of it, as well as the behavior and positions of other road users in the vicinity (such as pedestrians, cyclists, and other vehicles). Our engineers use this data to refine and improve the software to ensure the self-driving car performs safely.
So effectively, Waymo has secret process, which may change all the time, to decide which disengagements to report and which not to report. There is no way to tell whether the car would have slightly bumped into another car, or killed a child if not disengaged.
Their reports to the DMV do not constitute data, they are PR.
Not trying to single out Waymo, they all do that.
[1] https://web.archive.org/web/20190531071448/https://www.dmv.c...
Drivers driving without a valid license, without insurance, under the influence, wanted by the law, etc would have all kinds of reasons to not report a crash.
I also used to know a woman who thought it was nothing more than a fender-bender, went to the doc a couple of days later because her neck was bothering her. Doc sent her straight to the hospital--incomplete cervical fracture, one wrong move and she could have dropped dead.
In some cities currently even if you want to have police respond to a traffic accident, you'd need to wait a VERY long time - they will tell you that unless you need EMS to try and get a safe spot and call for a tow. Some folks aren't willing to wait hours for police to show up to do what, take a report?
> An estimated 15% of injury crashes and 24% of property damage-only crashes are never reported to police (M. Davis and Company, Inc., 2015), while9% of injury crashes and 24% of property damage-only crashes are reported but not logged (Blincoe et al., 2015). Even with robust data, establishing the statistical significance of automated vehicle safety can be expensive. Kalra and Paddock (2016)demonstrated that establishing that an AV has a fatal crash rate equivalent to the national average with 95% confidence would require driving a fleet of 100 vehicles continuously for 12.5 years.
I presume there's an adjustment for this in the figures you compare to, but I am having trouble finding it, though that may be my own failure so I wanted to ask if you could help me find how that's adjusted for?