Are you talking about bugs in artificial intelligence, or bugs in regular 'ol human intelligence?

Because that statement rings true for both, lol.

That's the real risk with self-driving systems. Bugs are sporadic. But design limitations create the risk that you could have mass-accident events if everyone on the road runs into the same edge case at the same time.

> Are you talking about bugs in artificial intelligence, or bugs in regular 'ol human intelligence?

> Because that statement rings true for both, lol.

Your stated position gives far too much credit to algorithms and far too little to what every capable human driver can do.

Far worse can happen, when it gets hacked.

The data available challenges this contention. According to here[0], one estimate is that there are approximately 18,000,000 people driving at any given time in the U.S. And according to here[1], there were 34,439 traffic fatalities in the U.S. for the calendar year of 2016.

For the sake of discussion, let's assume that of the 18 million drivers in the U.S. on the road at any given time, there are at least two other drivers which are not on the road at that same time. Further, let's assume these are all of the U.S. drivers for the entire 2016 calendar year.

This would put the estimated set of unique drivers at 54,000,000. A conservative estimate by any means.

This makes the estimated probability of a human driver being involved in a traffic fatality for the 2016 calendar year to be 34439 : 54000000, or 0.00063776.

According to here[2], the probability of being struck by lightning in the U.S. is approximately 1 : 280000, or 0.00000357.

While the former is mathematically 200x more probable, the latter is statistically insignificant.

> Capable human drivers as a subset of human drivers.

Hence my qualification of "capable", included in order to obviate edge condition "whataboutism."

> At least an av cant get drunk.

But it can get defective updates, be unable to identify unknown conditions, suffer from misidentification of threats, and susceptible to cascading failures affecting all autonomous vehicles in the same situations.

0 - https://www.quora.com/How-many-drivers-are-on-the-road-at-an...

1 - https://safer-america.com/car-accident-statistics/#US

2 - http://www.lightningsafety.com/nlsi_pls/probability.html

this should at least get you in the right direction, but you'll likely want to put motorbikes into their class at some point.

anyway short story long you want the cumulative chance of getting killed, basing the estimate on the average person lifetime miles, which varies widely anyway one person to another making a synthetic number over such variability unfit for any purpose

Okay, let's go through what you identify as being wrong point-by-point.

> first you don't need to normalize by r ruining vehicles, both metrics are yearly.

I provided the (active/daily) running vehicles metric in order to establish a (very) conservative estimation of 54m distinct human drivers in calendar year 2016. This was to ensure subsequent analysis was reasonably well founded by way of using common base units.

> second the real signal is death per mile driven, now that's something you should look for, not abstract drivers.

Perhaps that signal is significant for other premises, but mine was to provide a probability of any given human driver causing or being involved in a vehicular fatality. This was to ensure that the derived probability was reasonably correct based on referenced data.

> third using drivers as base assume all cars vehicles have no passenger.

Perhaps my use of the descriptor "driver" was improper for the entirety of the analysis. Given the limitations of the medium, it was a trade-off I was willing to make.

Your point is correct and, yes, it would affect the probability iff the intent was to determine driver liability.

> this should at least get you in the right direction, but you'll likely want to put motorbikes into their class at some point.

> anyway short story long you want the cumulative chance of getting killed, basing the estimate on the average person lifetime miles, which varies widely anyway one person to another making a synthetic number over such variability unfit for any purpose

Your analysis is keen and one which is goes beyond what I was trying to illustrate to the GP. To wit, the specific point being addressed by my terse analysis was:

> U give far to much credit to human drivers.

So I'm not sure that accidents per mile is necessarily the best metric either.

True, that estimation may be peak load. Which is why I kept the total estimate of distinct drivers at 54m instead of the much more likely number of well above[0] that.

0 - https://www.fhwa.dot.gov/policyinformation/pubs/hf/pl11028/c...

Secondly, you didn't actually provide a comparison. You simply remarked "Oh the chances of dying in a car accident is comparable to the chances of dying due to lightening therefor cars are safe" (whilst ignoring the 2 ORDERS OF MAGNITUDE difference between them, people struck by lightening != people killed by lightening and the fact that cars are not safe, as anyone who has ever driven will tell you).

https://news.ycombinator.com/newsguidelines.html

I provided data and analysis therein. You provided an emotional attack, and a vulgar one at that.

If you have data which supports your premise of:

  U give far to much credit to human drivers.

Here in Australia, with a population of 25 Million there where 1,140 road deaths in 2018.

The USA with it's 327 million population had some 40,000 road deaths on 2018.

I find it hard to imagine the av will end up doing any worse a job driving than the humans currently on the road.

And since, even with the occasional bug, the av would be doing a better job of it, that means they will end up saving lives (and lots of them).

Humans who cause accidents will have remorse and learn never to do it again, or be punished if alcohol was involved. With an AV, currently nobody knows. It could be as simple as "we've checked in the bugfix to prevent this from happening again" with no emotion behind it.

The reality is lots of people die on the road and the numbers show this sad fact.

Now the reality is some of those who are killed, die from no fault of their own. They die at the hands of the other driver.

The sooner we can take away that human element from driving the better, because that one act will end up saving lots of lives.

> Humans who cause accidents will have remorse and learn never to do it again

That is so untrue, at least for what happens here in Australia.

Here in Australia we have many hundreds in not thousands of repeat drink driving, speeding offenders on the roads, and short of locking them up there is no way to keep them off the roads.

Taking away their license makes no difference as they just drive without a license.

That's way fewer

The argument in this thread seems to suggest we should be slow at adopting av technology because it could kill some innocent people.

Now I accept that point as a fact. There is no doubt people will die in av cars just like people die in cars today.

Yet, on the other we have human drivers killing innocent people today, which is acceptable collateral damage, even though introducing av technology would help to greatly reduce that number.

Speaking for no one other than myself, vehicular deaths are not "acceptable collateral damage" so much as a statistical inevitability, one sought to be minimized as much as possible.

The problem with introducing autonomous vehicle technology, again IMHO, is the assumption that it "would help to greatly reduce that number."

This is not a known result.

And without actually doing the implementation that question will never get answered.

But logic suggests they would help only because, with the exceptions of bugs in the system, the av:

. would not drive drunk

. would not speed

. would not run red lights

. would not get tired

. has better reaction times than humans

. has better eyesight than humans

. is not distracted by mobile phones, passengers or pretty girls/guys walking on the foot path

. is not susceptible to road rage

. etc etc.

> And without actually doing the implementation that question will never get answered.

Exactly. Which is why it is imperative the engineering community does not assume outcomes beforehand.

Much of the concerns you list I agree are likely benefits, with the "not drive drunk", "not get tired", and "not susceptible to road rage" items being a near certainty due to eliminating biological aspects.

What autonomous vehicles might not be better at could be:

. snow

. black ice

. mud

. severe thunderstorms

. able to operate in high dust environments

. adapting to rapid unexpected operating conditions

. when to run a red light (emergencies, external threats, some combination of conditions above)

Are these edge conditions? Maybe. Or you could call them functional requirements not often discussed.

All I'm saying is that there is a lot more to general-purpose driving than what can be shown in a limited setting.

IOW, show me an autonomous vehicle which can complete the Paris-Dakar Rally[0] and I'll show you someone who will say it's time for people to stop driving ;-).

0 - https://en.wikipedia.org/wiki/Dakar_Rally

EDIT: formatting

> with the exceptions of bugs in the system

Being a software developer I have no doubt there will be bugs in these av software system.

But these systems will also include many fail safe overrides designed to minimize the impact of these occasional bugs.

That's BS. You can answer these questions with rigorous research (or a slow rollout).

You can't possibly think that drunk drivers and those driving without a license are the target market for self-driving cars.

You are correct, I don't think that.

That group just represents a small subset of the drivers that present a danger on the roads today.

Now since you asked, I would say the real target market for self-driving cars is big business.

They are the ones pushing hard for this technology, not for the safety aspects that I have focused on, but instead for the cost savings this technology will bring to their bottom line.

Easy to turn that around. Only individual humans learn from their mistakes - and even then, not always. But with AI, the whole fleet learns that lesson.

Yes, of course we want this to be a numbers game. What else would you have this discussion be about? Irrational feelings?

I mean, yes, it is fine for AVs to have an occasional bug --- as long as that occasional bug kills only kills one person per 100 million miles driven. But that's actually a pretty high bar.

[1] https://www.forbes.com/sites/bizcarson/2017/12/22/ubers-self...

Now if instead the road deaths stay the same or go up, then naturally that should should also spell the death of av technology.

The participants are in it for the profit, not for saving lives. The US health care misery is a prime example of what happens when you try to solve a situation through the market, which by definition has to treat human lives as an externality: People in the US die earlier than in any other first world country.

The causes of death might not be the healthcare system ,no? What if the majority of these deaths are caused by bad diets and stress? Would you blame the healthcare system for it?

You're right, one can't isolate one institution for topics as complex as these. I'm pretty sure it plays a role, though, as (ignoring fundamental differences in poverty rates, workplace regulations etc.) many countries employ the healthcare system to inform the population about healthy eating and offer incentives and free courses for coping with stress. Thats the beauty of a single payer system: Beyond ethics, the healthcare system has an intrinsic interest to prevent the preventable, as that is almost universally cheaper than bearing the cost of circulatory disorders later.

There is no data to suggest that this will happen. Furthermore, if self-driving cars do take off, they will continue to be outnumbered by normal cars for a very long time. It's not even clear that they will or should be the majority.

Humans making a reckless mistake while driving go to jail.

Are those that advocate the promise of this tech willing to say "Reckless commits and reckless code reviews send engineers to jail"?

We all know that the fastest and most reliable way to find and fix bugs is to implement the thing in production and see what happens. As you point out, it exposes problems that testing and design just can't.

I see people here obsessing over how much will change if we had fleets of AVs. Maybe shaving five years off the wait time is worth N number of human lives. Especially if N < the number of lives will ultimately be saved if the technology is perfected years earlier.

Of course I wouldn't want myself or my loved ones to make that sacrifice. I don't want to die in a regular car accident either. Given the upside, speeding up AV development may be worth the risk?

This would set the incentives straight at least.

For every demand they make of OTHER PEOPLE to improve the safety of travel, they are not willing to do even /10 of that.

Not if you wish to win the hearts and minds of the general population at large. As grandparent points out, the coverage seems disproportionate, but that's because it interest people, and they are generally sceptical of letting computers drive their cars.

While technically unrelated, the Boeing 737 MAX MCAS scandal hasn't done much good either for the further automation of transport systems.

People dying at the 'hands' of AVs will lead to pushback from the general population, and political points can be scored by siding with this view, leading to tigther regulation if not an outright ban some places on AVs.

The real question is; will either the slow and thorough approach or the quick and dirty approach roll out larger AV fleets faster than the other? In any case, it'll probably be decades, before there are significant AV fleets.

But the car was such a valuable and exciting technological leap that people just lived with the risks.

It'll be the same here too, if the technology is valuable and life changing.

Meanwhile 1.25 million people die in human caused car accidents every year and somewhere around an additional 30 million are injured. But that’s “just the world we live in.” Or that’s “your choice to take a risk when you choose to drive a car.” Or whatever.

A lot of criticism of our industry is well-deserved if overly broadly applied. Most companies do not operate with the total lack of ethics that Uber and Facebook do. But they are so big and so well-known, they’ve come to symbolize a culture of amorality, deception, manipulation, and a willingness to do anything for growth and fresh injections of VC funding.

What would be really useful for everyone is to throw out that narrative for a while and focus on the task at hand. Self-driving cars will be regulated. There’s no getting around that. And tech companies could help themselves out a lot by accepting that and coming to the table with regulators with an actual plan instead of whining about having their innovation stifled.

There is a reasonable compromise. You can’t start by saying the error rate must be at zero. That will never get anywhere and people will keep dying a million a year. Where is the right number and what does it measure? I don’t know. It should probably be tiered by population density and average driving conditions and weighted by number of annual deaths in that city/region.

Highways are an easier problem to solve than, say, Manhattan. But highway deaths are more likely to be fatal. So the standards ought to apply differently.

The challenge here is really to get regulators to think a little bit like problem solvers rather than enforcers for a change. To get them to realize that “regulate self driving cars” is a problem so big and mushy that it’s pretty much meaningless. But “establish safety standard for interstate travel for autonomous trucks” is something you can work with. Maybe it’s 100x as a safe as a human driver in that scenario. I don’t really know.

Tech companies could probably turn on a lot of positive PR if they promoted milestones for certain communities. Maybe even offered incentives for cities to improve road markings. Like if Waymo said, “We want to target Brooklyn, and our goal is 10x as safe as human drivers in Brooklyn with an additional target of 1000x fewer cycling accidents. But we need you to mark the bike lanes better and put them between the parked cars and the curbs where it’s feasible, and here’s a pile of money you can use to do it.” That would generate a lot of good will, positive pr, and maybe even genuine enthusiasm that they can use to open up that market for their taxis ahead of schedule.

The car companies will get a lot better treatment from regulators if they chunk the problem into small pieces and go to the negotiating table without a victim chip on their shoulder.

There was a nice video of a Tesla driving at low speed constantly pulling the car towards green islands between lanes as if it had a death wish.

Uber famously killed someone while testing and then blamed the humans.

It remains to be seen how Waymo will fare, although having the advertising company that's spying on everyone build self-driving cars seems like parody.

Using Tesla's very clearly level-2 system, or Uber's rubber-bands-and-coathangers science project, to criticize Waymo isn't particularly fair either.

If we set up the goalposts fairly, Waymo wouldn't be anywhere near receiving permissions for full autonomy until we had statistically significant evidence the latest build could reduce fatalities to human levels which are in the region of 1 per 100 million miles (a rate which includes all the people who aren't allowed on the road...). When they've only driven a few million miles, the fatal error rate - including potentially fatal collisions avoided by safety drivers - really should be zero.

We haven't got any evidence that fatal errors in AV technology approaches that level of rarity, will never have it for individual builds, and the best [limited] evidence we've got refuses to reject the null that despite being shielded from hard AI problems like left turns and supported by safety drivers and driven at low speeds, AVs are an order of magnitude more deathly than human-driven cars...

An AV company should only be able to update the car software after that software has correctly driven for X million miles.

Perhaps a separate agency should perform the actual updating (installing of the new software), so we can be certain that no "quick fixes" are happening behind the scenes.

When we use automatic vehicle, we assume it to be a reliable driver, not a random person who could be a bad driver.

So please stop comparing to average human drivers.

Also, software gives the possobility that the same shitty driver can drive millions of cars at the same time. One bad commit might cause a disaster. Millions of human drivers wont be possesed by the devil at the same time causing havoc. Imagine all Toyotas turning full left at the same time.

I guess this is your informed opinion that comes after reading about Tesla's autopilot crashes. And I share that opinion.

But I have the feeling that Waymo's system is much more mature than Tesla's. Let's see.

No way. Not even close by any factor. Tesla have real data from real people driving real cars across the world. On the other hand, Waymo have data only from their Test vehicles and augmented scenarios.

https://electrek.co/2018/11/28/tesla-autopilot-1-billion-mil...

It’s going to be extremely hard for AV manufacturer to explain that it’s impossible for them to guarantee that a certain abnormal autopilot behavior that caused an accident isn’t going to happen again.

We had an entry in the DARPA Grand Challenge in 2005. It was too slow, but it didn't crash. We profiled the road ahead with a LIDAR, so we had an elevation map of what was ahead. This is essential for off-road driving, and it's the gold standard for on-road driving.

But there's a range limit. Not from the LIDAR itself. From the geometry. If the sensor is 2m from the ground, and you're looking 30m out, the angle is so shallow you can barely sense elevation. You can't see potholes. Anything that looks like a bump hides a lot of road behind it.

The Stanford team's answer to this was to use machine learning to compare the near road texture, which they could profile, with the far road texture, which they couldn't. If the near road was flat, and the far road looked like the near road, they could out-drive their profiling range. If it didn't match, slowing down brought the stopping distance down to profiling range, and they could work through difficult terrain. The 2005 Grand Challenge didn't have much difficult off-road terrain. No rock-crawling or cratered roads. The 2004 course was harder, and nobody got past mile 7. So most of the time, fast mode was usable.

Google/Waymo started from the Stanford approach, trying hard to profile the immediate surroundings. Their machine learning efforts were focused on identifying other road users - moving objects, not fixed ones. Their earlier videos make this clear.

Google/Waymo built a cute little bubble car with a top speed of about 25MPH and a LIDAR on top. At that speed, you can profile the terrain all the way out to your stopping distance, so you have solid detection of fixed obstacles. That was something that had a good chance of working well. They decided not to manufacture it, probably because it would cost far too much.

Machine learning isn't that accurate. You can get to 90% on many problems. Getting to 99% is very tough, and getting to 99.9% is usually out of reach. The killer apps for machine learning are in low-accuracy businesses like ad targeting. Or in areas where humans have accuracy in the 80%-90% range.

Here lies the problem. Humans are very good at obstacle avoidance. Much better than 99%. Machine learning isn't as good as this problem needs.

There's another side to the problem - the false alarm rate. If you build a system which insists on a valid ground profile, a piece of crumpled cardboard on the road will result in slowing down until the sensors can look down on the cardboard and see past it to solid pavement. You get a jerky ride from a conservative system. That's why Uber disabled automatic braking and killed a pedestrian. That's why Tesla's system fails to react to fixed obstacles it could potentially detect. Waymo has struggled with this. Customer evaluation of driving quality seems to be based on good lane-keeping and low jerk. These are things that trouble poor human drivers. Self-driving has different strengths and weaknesses. This is what leads to systems which seem to be doing great, right up until they crash.

What self-driving seems to need right now is a rock-solid way of detecting the lack of obstacles ahead. All we have so far is LIDAR. Radar is still too coarse and has trouble distinguishing ground return from obstacles. Even LIDAR is rather coarse-grained. Stereo vision doesn't seem to be hugely successful at this. We need that before self-driving vehicles can be trusted not to run into obvious obstacles.

If you have to recognize what the obstacle is before determining that it's an obstacle, it's not going to work.

There are a whole series of secondary problems, from left turns to double-parked cars. But those are not the ones that kill people. It's the basic "don't hit stuff" problem that is not adequately solved.

[1] https://github.com/John-Nagle/Overbot

(That's part of the price of hostility to software patents - more trade secrets.)

I'm guessing the answer is currently no. Which is interesting because one of the early benefits touted for self-driving cars was that people might die, but a patch would ensure no one dies twice for the same reason (within reason, which is more than can be said for humans).

Companies frequently misrepresent how their technology works to the public. Typically they'll do some small portion with the hot technology for buzzword compliance, and then build the rest of it with an actually sensible, boring, and tailored-to-the-problem technology stack, oftentimes with a lot of proprietary legwork done by their data scientists and engineers. This way they get the best of all worlds: investment dollars from gullible investors, PR from journalists that want to hop on the next big thing, a product that actually works, and misdirection so competitors hop on approaches that aren't going to work anyway.

My original point was that the decision tree should hopefully degrade gracefully as the prediction quality goes down, and not have any brand that leads to an insane behavior.

Now obviously, the better your predictions, the better the driving will be, which is why you'd want NN accelerators.

I want to assume this too, but what evidence is there that we can actually assume this?

I don't really understand how this meme gets so much traction. It just ignores the enormous number of miles driven by Americans every year. Last year, Americans drove 3.22 trillion miles. There were about 6 million accidents, or 500,000 miles between accidents. (Or about 37 years of driving at typical rates). Driving results in a death every 85 million miles traveled.

Driving is actually very safe. Conversely, because it is so safe, it wouldn't take much to dramatically increase accident rates from driving.

  I don't really understand how this
  meme gets so much traction.

For example, tell someone "cars kill as many people as twelve September-11th attacks every year" and they might wonder where the trillion-dollar war on road deaths is.

On the other hand, tell someone "heart disease kills 17 times as many people as cars every year" and you'll give them the opposite impression.

Sure, our perception of risk and regulatory responses are not statistically accurate. Commercial air travel is much safer than cars, yet regulatory resposnses to risk are stricter. Is that a bad thing? It creates a very safe mode of transport.

Going driverless is a new mode, and an opportunity to create a safer future. Why not set a standard that is much higher than human operated cars?

Also, a lot is unknown. We won't know what driveless' safety record is until it truly exists in the wild.

The economic pressures to go driverless will be immense^. Meanwhile, a high safety standard could require a some counterpressure, and the starting point will set the stage for the future.

^two kinds. It could be a huge win for consumers, with taxis prices halving or better. It is alsi an opportunity to "put capital to work," in a (current) market that is starved for such. A robo-taxi fleet is just the kind of thing a massive fund can erm.. fund.

> I hope we can view these incidents in context.

The desire to hold some entity legally responsible for any accidents seems to be the key problem. That problem is really about punishing the responsible party. If someone dies because of an irresponsible driver, the outcomes of the case are always some sort combination of punishments: fines, incarceration, suspensions, etc.

Proposed Solution:

Legislation can easily be passed that provides clear guidelines about how these errors of judgement in computing systems can be punished. For example:

In situations of death, $X is paid to the victim's family, $Y is paid to the federal government, the company's automated vehicle manufacture license is suspended until federal review, etc.

Then, its a simple matter of getting humans to sign to agree that they will accept these terms.

Kind of off-topic, but do these minivans have insurance policies? Or is there some Alphabet subsidiary that allows them to act as insurance for themselves?

If they do basically insure themselves, they will already be paying out insurance whenever their AI fails to follow/obey laws, so the only issue with placing blame is in the public/media's eye.

minor nitpick but it's about from which pocked compensation comes out, not really punishment.

It's very dangerous in some places in the world. Majority of deaths are happening in places with poor infrastructure[1], that autonomous vehicles are very far away from being able to handle. For current state of AV, bar it has to meet and exceed are places with good infrastructure, fairly new cars and reasonably good driving culture, as those are first targets of rollout.

For example, in Phoenix, AZ, about 250 people die per year from accidents[2], with about 9k miles driven per person[3] and metropolitan area having 4.7m people, to just match human levels AVs to have less than 1 fatality per 170 million miles driven.

[1] https://en.wikipedia.org/wiki/List_of_countries_by_traffic-r... [2] http://www.city-data.com/accidents/acc-Phoenix-Arizona.html [3] https://arizonapirgedfund.org/sites/pirg/files/reports/FINAL...

In other words, it's still more like a better cruise control: it makes the easy part of driving easier, but doesn't help with the hard parts (and may make them even harder.)

Explain the universe where humans are rational enough to give "proportionate" attention to tech as novel as self-driving cars, yet irrational enough to have preferred private car ownership over mass transit in the first place.

Cabs are SO convenient, and public transportation has all kinds of issues world-wide. Buenos Aires, for example, has subsidized metro (about 0.50U$S per ticket) vs a non-subsidized uber (5U$S per ride~) and still the experience is so much better and convenient and reliable that 5U$S is really worth it.

I think that the ride-hailing revolution is a classic case of great microeconomics.

Every time a self-driving car causes an accident in a situation where a human driver probably would not have (and sometimes in pretty obvious ways, like driving into a freeway divider), it will hurt acceptance of the new technology. Politicians will never miss an opportunity to grandstand against tech companies. People will be afraid to ride in autonomous vehicles.

Autonomous vehicles are dangerous in unpredictable ways. People might be injured or die in accidents that would not have happened with human drivers, as the result of software bugs rather than human decisions. When things like that happen, statistics about accident and death rates being lower with self-driving cars are beside the point.

Its politicians, who are all about "click-bait-emotionally driven" to bring them votes, you should worry about.

If you're concerned about the costs of a driver, I grant that the marginal costs per hour of driving is low, but the material costs seem high, unless a significant cost breakthrough is found in lidar production, or a technique breakthrough using multiple cameras; and the R&D costs seem pretty enormous -- people have been seriously working on this since the 80s and it's clearly closer than fusion, but it seems perenially 15 years away.

Everyone likes to think they're better than average (https://en.wikipedia.org/wiki/Illusory_superiority#Driving_a...).

I would rather keep the worst drivers off the road than keep the best.

A machine suddenly hitting a bug is whole new ball game though. I guess over time maybe you could build up a sense of where machines have trouble driving and adjust accordingly.

That isn't just a question of the safety of the autonomous systems though, it also includes consideration of which drivers end up using them. Some drunks will use auto taxis, lots won't.

I still think marketing it as a safety issue was the wrong approach this early in their development. Say "safety" and "cars" in the same sentence and people think of seat belts, air bags, and anti-lock brakes. All things to protect you from an accident on unpredictable situation.

If you're in the latter group, you would be rightly horrified each time there's an AV accident.

(the "on average across the population it will be safer" doesn't mean anything for drivers who are already well above average)

Besides we need solution that works for 80% of the people. there will always be some edge cases

I view them in the context "software can kill people, and can do so at scale." Same way I view the recent Boeing 737 incidents.

Ultimately it will come down to, who is safer?

No two humans have identical brains, so every human-caused accident is unique.

In other words: human errors do not scale at all, fortunately.

Worldwide there were over 1.3 million road deaths in 2016. That’s a pretty big scale.

Technology, even when deliberately employed to cause death, has rarely managed to cause such carnage.

Not to say that self-driving cars will be ridiculously unsafe when the first commercial service appears, but they share one thing with planes: increased safety is an engineering problem.

Every incident that occurs with these vehicles will have tons of data and people looking over that data to fix the issue. Human brains can't be fixed, sensors and computer software can.

Airplane manufacturing is extremely highly regulated - the reason being that if there's a bug, then the bug is in every airplane. And when that bug appears there's often very little opportunity to mitigate it and the consequences are huge, it's the Manufacturer that's responsible, but the passengers (and bystanders) who pay the price.

Similarly, the standard for safety in car manufacture is much higher than the standard of safety for becoming a driver. Partly because the problems for the former are systematic, and partly because the latter primarily poses a risk to themselves. There's self-regulation there.

When Tesla tells everyone they have autopilot it isn't Elon Musk whose Tesla slams into a lane divider at 70mph, it's one of the customers he lied to, and it's likely not even one customer, it's a systematic issue where every driver in that situation will face danger.

I'm not aware of any fallback remote takeover mechanism that can be used to recover AV's on this situation.

EDIT: I'm convinced, that until we majorly overhaul our road infrastructure to be more 'robot friendly', it's hard to see _fully_ autonomous AV's work in major cities I.e. electronic beacons on lane dividers, electronic signals on traffic lights, vastly stronger GPS network or replacement thereof, six nines level mobile network coverage at high bandwidth.

Insurers will be a major factor here. They should start getting some pretty good insights in accident rates involving AVs vs. normal cars.

Good luck affording even basic collision avoidance tech in a family car if you are an unemployed driver.

That is extremely unlikely and borders on impossible. AVs have some profound things working against them. They share the problem airplanes have. Human beings are abysmal at risk estimation, and a very large factor of the estimation they do is whether they have personal control. It doesn't matter that planes are much safer than driving, the fact that people don't have direct control of the aircraft makes them feel unsafe to the point that some people are paralyzed by their fear of flying. The same will be true of AVs, and for the same (un)reasons.

But I have my own doubts that it will even get far enough for that to pose a serious problem. We have a more general problem with technology in our society. It is nearly unanswerable to the law. Take the Toyota "unintended acceleration" scandal from a few years ago for example. Investigation of that issue determined firmware flaws were the cause, and led to the deaths of over a dozen people. The criminal court case against Toyota unearthed that while the automotive industry had coding practices that established 'required' and 'suggested' techniques, lists of around 100 practices. Toyota followed 4 of them. Firmware developers working for Toyota did not even have access to a bug tracker at all. They did not have access to static analysis tools. They had no role in determining scheduling.

They were found 'not guilty' for criminal negligence. The reason given by the court was that because the problem involved software, criminal negligence was impossible. No standards exist governing software, and therefor no company can be held criminally responsible regardless of their practices. During the civil case, they were found guilty and responsible for the deaths, but before the jury could award damages, Toyota settled with the victims families.

That legal precedent is terrifying. The first AV to market will inevitably be the one that cuts the most corners and is rushed the most in its development. When it kills someone, as is basically guaranteed to happen by simple statistics, the public will be paying more attention than they did to the Toyota case. And when the public at large hears that tech company executives are literally above the law, the outrage and consequences will likely be swift and severe. Sensing political opportunity, the legislators will descend and could very well end up dictating what language must be used, what coding standards must be followed, what tools must be used (you know the tool vendors will be lobbying hard), etc. The ACM (Association of Computing Machinery) has been discussing this issue for years, debating the potential merits and pitfalls of establishing a licensing system for software engineers. Thus far, the discussion always ends with 'companies would have to pay more for licensed developers and the cost of labor must be contained.' The looming pitfall of avoiding certification and licensing, however, is that government will step in and do it themselves which is virtually guaranteed to enact the greatest pain on the software industry.

Granted, you're still much more likely to die of heart disease. But if we're going to care about any non-disease causes of death or injury at all, automobile accidents are near the top of the ones to worry about. It's a much bigger cause of both deaths and injuries than violent crime is, for example.

AVs certainly have bugs, and some may lead to accidents and even deaths, but each one will be examined in an enormous amount of detail to figure out what sort of edge case it was and prevent similar ones from happening.

The context is that Tesla hardware is not sufficiently powerful, so they do "optimizations" and ignore stationary objects. This is not a video game where we can trick the player with some clever hacks, if the hardware is not fast enough the solution is to use better hardware and I think is very normal to see in newspaper when "autopilot" hits a wall, it is a public service even if Tesla fans and shareholders don't like it and try to play with numbers to show some statistics to favor Tesla(AFAIK Tesla still keeps the safety numbers they were using to promote their cools stats a secret)