Also, not all miles are equal. I'd hope that people are enabling Autopilot primarily while driving on straightish roads with good driving conditions -- in other words, at times when the rate of fatalities with a human driver is far less than 1.3 per 100M miles.
[0] http://safety.fhwa.dot.gov/speedmgt/data_facts/docs/fataltbl...
The only measure that makes sense in this context is time, bringing distance into it is statistical trickery.
If you expose yourself on a live fire range for 1 second and travel 100m/s, it's far less dangerous than exposing yourself on a live fire range for 100 seconds travelling at 1m/s.
It's the same thing with cars, the longer you are physically on the road timewise, the longer you are exposing yourself to the chance of an accident, regardless of distance travelled.
Not sure why you think it's "absolute", you haven't listed a single reason or argument, just a tautology.
Conversely, when you hit something on the highway, it's likely to be moving much more slowly than you (e.g. the chicken crossing the road), or at a speed similar to you (the drunk guy driving on the left side). So your speed matters - you're more likely to reach that obstacle in any given period of time if you can explore more space in that time.
If my job is 2 hours away because there is no motorway, I will get a different job. If there is a motorway, making the commute 40mins, I will take that job.
People care how long the journey is, not how far the journey is.
People live by commute time, not commute distance. Same for service areas, if the roads all have low speed limits you would need more drivers, they wouldn't just drive for longer.
Accidents per hour of driving would be more useful for public policymakers – given fixed average commute 1 hour per day, what's the tradeoffs of building highways (worker mobility, commerce, pollution, landscape, accidents).
And most of the highways that don't have a speed limit are quite narrow so while it is possible, no sane driver would do so.
If you look at highway A5 as an example. This highway has 8 lanes, which makes driving fast quite dull since you can not really feel the speed relative to the environment.
Outside of EU, the only other European location that would qualify would be Isle of Man (as referenced in that article).
For those familiar with the UK road network see: http://roadsafetyfoundation.org/media/32639/rrm_britain_2015...
I know when I tried my friend's auto-pilot that I was concentrating 10x more on the road than I would be normally. Obviously if I got more comfortable with the tech then I would be concentrating less but I can't imagine that I would ever stop paying attention unless I knew the system was fool-proof.
This must be harder than concentrating when you're actively engaged?
"Given that current traffic fatalities and injuries are rare events compared with vehicle miles traveled, we show that fully autonomous vehicles would have to be driven hundreds of millions of miles and sometimes hundreds of billions of miles to demonstrate their safety in terms of fatalities and injuries. Under even aggressive testing assumptions, existing fleets would take tens and sometimes hundreds of years to drive these miles — an impossible proposition if the aim is to demonstrate performance prior to releasing them for consumer use. Our findings demonstrate that developers of this technology and third-party testers cannot simply drive their way to safety. Instead, they will need to develop innovative methods of demonstrating safety and reliability."
For near-misses, disconnects, poor decisions, and accidents, they replay the event in the simulator using the live data, and work on the software until that won't happen again. Analyzing problems which didn't rise to the level of an accident provides far more situations to analyze. They're not limited to just accidents.
See Chris Urmson's talk at SXSW, which has lots of playbacks of situations Google cars have encountered.[1]
Note that the number of crashes per 100 million miles is a lot bigger than the number of injuries. One would hope a statement about safety would look at all of the data.
PS: Beside should one more fatalities occur soon, by they're own standard, Model S would become statistically more deadful. To much risk of backfire, this PR is madness...
Before the accident occurs, these numbers were 1/100B miles and 5.3/100M miles.
If a second fatality occurred, those numbers would increase to 3.4/1B miles and 9.3/100M miles.
However, I'm pretty sure that their claims of safety are based on more than just this calculation: "...a wealth of internal data demonstrating safer, more predictable vehicle control performance..."
100M miles of driving is easily enough time to observe things like fewer near misses, greater distance between the Tesla and other cars, etc. In math terms, what Tesla is most likely doing is working off some model relating driving behavior with accident rates and observing that 1 single accident does not invalidate this model.
This is from http://www.rand.org/pubs/research_reports/RR1478.html
I'd love to hear any problems with RAND's analysis, because it'd be great for this to not be the case.
But each of those assumptions is problematic at best. It's a variant of the "Soldiers in Vietnam" problem. In 1961, the U.S. had 3,200 soldiers in Vietnam, and only 16 of them died. If straight extrapolation worked, you could infer that in 1968, when troop levels peaked at 536,000, we'd be looking at 2,680 deaths.
Actually, 16,899 American soldiers died in 1968. The mortality rate climbed 6x, because the war became quite different. American troops were exposed to all sorts of lethal situations that weren't part of the mix in 1961. Data is here http://www.archives.gov/research/military/vietnam-war/casual... http://www.americanwarlibrary.com/vietnam/vwatl.htm
In Tesla's case, it's likely that a lot of the test miles so far have been incurred on California freeways or their equivalent, where lanes are wide, markings are good, rain is rare and snow is unheard of. Start moving more of the traffic load to narrower roads where nastier weather is common, and you're adding stress factors. Demographics change, too. The high price tag of Tesla vehicles today means that most drivers are likely to be in their prime earning/wealth years (let's say 35 to 70.) If the technology becomes more ubiquitous, we'll need to see how it performs in the hands of people of teenagers and the elderly, too. Once again, the risks rise.
Again, please note that all I'm saying is that the secondary metrics are the primary predictor here since they are statistically significsnf. The primary metric is just an order of magnitude check since we don't have enough data (note the width of my posterior) to do more than reject the secondary metrics if they are crazily off.
https://www.chrisstucchio.com/blog/2013/bayesian_analysis_co...
Finding that the lower end of a credible interval close to zero is an order of magnitude below the top end is pretty common. It just means you need more data, that the beta hasn't approached a gaussian yet.
In [1]: from scipy.stats import beta
In [2]: from numpy import percentile
In [3]: data = beta(1, 100e6+1).rvs(1e7)
In [4]: percentile(data, 0.5), percentile(data, 50), percentile(data, 99.5)
Out[4]: (5.0460240722165642e-11, 6.9295652915902601e-09, 5.3005747758871231e-08)Back to basics: If you assume that each mile is uncorrelated, (on the large not a crazy assumption), then the overall number of accidents over N miles can be modeled by a binomial distribution with N trials and some crash probability p. You're trying to use bayesian analysis to estimate that p.
The core concept there is that the probability of the observation given a certain p is directly proportional to the probability of the p given a certain observation.
Given p, it's trivial to compute the chance of the observation; just plug it into the pmf. For exactly one crash (and a small p), you can even quite accurately approximate it by Np/exp(Np) due to the trivial binomial coefficient for k=1 and the taylor expansion of the base-near-1 power.
So regardless of the numerical analysis that lead you to believe otherwise: you can just plug in numbers can get a numeric approximation of the (scaled) distribution of the probability mass for p. When I do that, both with an exponential prior (i.e. p in 0.1-0.01 is as likely as p in 0.01-0.001) or a uniform prior (i.e. p in 0.1-0.2 is as likely as p in 0.2-0.3) I get the unsurprising result that the maximum likelihood estimation for p is 1/N, and that the median p is a little lower; at around 1 accident in 1.5e8 miles
In practice that means that common sense is correct: if you observe 1 accident in N miles, a reasonable approximation of the best guess of the true accident rate per mile is 1/N.
Now, you can do this analytically, and the conjugate prior of the binomial is indeed a beta distribution. But that forces you to pick parameters for the prior from that beta distribution, and it's easy to do that wrongly.
A reasonable prior might be Beta[1,3] which assumes p is likely closer to 0 than 1. Analytically [then](http://www.johndcook.com/blog/conjugate_prior_diagram/), the posterior distribution is Beta[1+1,3+1e8-1]; with a mean p of 2e-8 (but note that the details of the chosen parameters definitely impact the outcome). This is quite close to the numerically derived p, though probably less accurate since it is constrained to a unreasonable prior.
So I'm not sure exactly what your python script is computing, but commonsense, numerical analysis, and Bayesian analysis all arrive at roughly 1/N in my case - you probably made some mistake in your reasoning somewhere (if you elaborate in detail how you arrive at this numpy snippet, maybe we'll find the error).
Note that this is not particularly sensitive to whether you choose beta[1,1] or Beta[1,3] as the prior.
Is there some principled reason to pick one over the other, or is it just "whatever looks reasonable"?
I also can't explain why plain numerical simulation doesn't come up with almost identical p values - I get 1/1.5e8 to 1/1.7e8, but that's 6.25e-9. I mean, it's the same order of magnitude, but it's nevertheless quite different.
Oh well, unless you've got some insight there, I guess I'm going to give up on that analysis - that kind of nitty gritty stuff is a timesink ;-).
This is also the company that thinks level 4 self-driving can be done without LIDAR. They're idiots, frankly, completely ignoring the difficulties of computer vision (both in computation and optics) that will not be overcome anytime soon. Even their fatality seems to have been rooted in their camera's inability to distinguish the color of the truck from the color of the sky.
Tesla hasn't even taught a car to navigate an intersection on its own, much less all the things Google's car is capable of these days.
I think what all this shows is the importance of regulators getting off their asses and establishing standards for self-driving vehicles. Perhaps the fatality was the event that will get them in gear. Poor guy.
They probably mean that 1 occurrence is not enough to invalidate a statistical model built on wealth of other data.
http://www-fars.nhtsa.dot.gov/Main/index.aspx
It has data from 1994 to 2014, including this plus several other statistics. In particular, it looks like they are tracking on the order of trillions of miles driven per year, so you're right, making any kind of statement after only 100 million is more shameful marketing than anything else.
Edit: this is US-only, while Tesla is claiming vs. worldwide. Clearly more miles driven worldwide, and probably higher deaths per 100m miles.
The proper comparison would be with non-Autopilot Tesla miles. But that comparison makes Autopilot look bad.
Remove the mileage driven from the cars that have never used autopilot, to control for drivers too cautious to trust their lives to beta software, and you should have your demographic.
You raise whether there is statistical significance about which is safer, autopilot or a typical driver. However it's worth noting that in a lawsuit the burden of proof is on the plaintiff.
So at best they matched what a human alone can do. Meaning the autopilot has done nothing at all.
https://en.wikipedia.org/wiki/List_of_countries_by_traffic-r...
>Seventy-four per cent of road traffic deaths occur in middle-income countries, which account for 70% of the world’s population, but only 53% of the world’s registered vehicles, burdens for 74% of world's road deaths. In low-income countries it is even worse. Only one percent of the world's registered cars produce 16% of world's road traffic deaths.
i also stumbled over this statement, but i think they are using all telemetry data (e.g. what would the autopilot have done vs what the human actually did) in all accidents, not just fatalities per miles driven.
I think this is a foolish statement: fatal accidents in the US are 1.3 per 100M miles driven. Tesla reports autopilot has driven 100M miles. That's not enough data collected to draw this kind of conclusion.
There's a parallel here with software testing. I've seen bugs that happen 25% of the time, for example, and take 10 minutes to run a test. Our test guys have great intentions, but if they test 5 times and see no bug, they think the bug is gone. There is no instinctive understanding that they have insufficient data to draw a conclusion.