Let's say everyone replaced their current cars with self-driving vehicles and they never killed anyone ever again. If normal vehicle usage continued and fifty years passed without a single traffic death, your claim suggests we'd still be unable to say if self-driving cars were safer, as it would still be a sample size of one.

I don't think your use of statistics is correct.