With autonomous cars, even once you have a few years of safety data from a large enough number of cars to be able to make the call of them being 5x less dangerous than human-driven alternatives in that data, you will still end up having much more "unknown unknowns" (of which I can't tell you any, because they are by design unknown) in addition to also having much more "known unknowns" like the possibility for large-scale software bugs potentially causing thousands of casualties at the same time. These risks will only go down slowly with enough time, there's practically no way of fast tracking getting these down, hence you have to incorporate a large enough risk buffer in your assumptions for rationalizing to even start using that fancy new tech, and the only place where this risk buffer can come from is having a much bigger difference in the "known knowns" department of risks.
Nothing. That's why regardless of how impressively big these test fleets are, there will be a lot more of these unknowns.
As soon as self-driving cars become a regular part of people's lives and not an exciting new thing, the calculation will shift to a much more rational one