Disengagements is a useful number, and the disengagement numbers say so much more about the real state of self-driving cars: That they fail as often as a non-synthetic oil change. And if humans had driving system failures that often, we would not be considered good at driving.
The longest number a company should be marketing is how many miles they've driven between disengagements. Like the "days since an accident" sign at a workplace: Fail, and you go back to zero. The numbers will be smaller, but they'll be a lot more meaningful.
i'm not sure what benefit there would be to gaming the disengagement stats. If anything, i think the incentive would be the other way around - inflate your number of disengagements so your competition thinks you're further behind than you actually are.
The point of testing is to bring a product to market, and if your product sucks people are going to figure that out relatively quickly. hiding your disengagements from the california DMV isn't going to do anything for you except build hype your product can't live up to.
Risk also skews results: Pilots taking off and landing on beaches in Alaska aren't comparable to pilots taking off from airports on the basis of incidents per 1000 hours.
It all depends on the content of the miles. And without transparency about that it sounds impressive but could very well be 95%+ that doesn't contribute much and where your average toddler without a driving license could have handled the car.
Why? I'm as curious as anyone, but what is the argument for mandatory disclosure of disengagement numbers?
TLDR: Cruise had, what some would consider, to be a disengagement incident but didn't report it because it does not fall into the categories defined by California.
The core issue here is that the disengagement reports should not be the end-all-be-all of where companies are. As the ieee article calls out, we have no clue what kind of testing the companies are doing (how hard they are pushing their systems). There is also some room for companies to avoid reporting certain types of incidents.
Waymo did 2 million miles in 2017, but only 330k were in CA. We are only seeing a slice of their data.
If we (the public) want a better picture of where companies are on this, the rules around what is reported need to be changed. They possibly need to also be done at the federal level so no company can hide their public testing in states that don't require reporting.