This happens to human drivers every year. Snow, freezing rain, fog; these adverse weather conditions can lead to large numbers of collisions at the same time. Expecting self driving cars to never have a problem during rare environmental corner cases is to expect them to be vastly superior to human drivers all the time. If you're waiting for that day before you consent to self-driving cars sharing the road with you, you'll be waiting a long time.
Cars sliding out of control on a snowy road is not a systemic failure.
If you have an entire fleet of self-driving cars using the same (or similar) technology, the exact same condition could cause all of them to misbehave in the exact the same way. This could happen simultaneously with a large number of cars. Or could be something that propagates in a cascading manner. (A car breaks, turns and spins out of control. The next car detects that, breaks, turns and spins out of control, etc.)
If you have no clear idea how your driving software works, how are you going to make sure this doesn't happen?
People have common sense, variability and adaptive learning to prevent things like that. Moreover, our traffic system, our cars and our driver education have been co-evolving prioritizing safety for the last 100 years. (Heck, some of it started with horses and carriages.) The complexity and size of the system have been gradually scaled up for just as long, and it was designed and refined to be used by humans.
It's astounding how many of AI enthusiasts fail to see these issues and differences. Traffic system isn't a board game.
A lot of people don't know how typing "www.google.com" into their browser works but DNS "just works" for them. There is software in nearly every part of our lives that could systemically go wrong in catastrophic ways - we do our best to plan and prevent them.
> People have common sense, variability and adaptive learning to prevent things like that. Moreover, our traffic system, our cars and our driver education have been co-evolving for safety for the last 100 years. (Heck, some of it started with horses and carriages.) The complexity and size of the system have been gradually scaled up for just as long, and it's designed and refined to be used by humans.
Isn't that incredible? The complexity goes up: we adapt. As self-driving cars are introduced we adapt and evolve and incorporate these changes in technology, the driving "environment", security, safety measures, and laws and regulations.
> Traffic system isn't a board game.
I didn't see any efforts at self-driving vehicles just say "ahhhh! this driving stuff is so easy!" Of course it comes with huge difficulties, an enormous amount of complexity, a baffling amount of data, a ton of edge cases, and now a wealth of tribal knowledge on the subject.
When considering these things think of the systemic benefits as well. There was a story on the Google self-driving cars in Mountain View not taking a turn when the passengers of the car expected it would. Turns out that the IR sensors on the vehicle saw that a jogger behind a wall of hedges was about to enter the intersection to cross while jogging. Systemically implementing that change across a fleet of cars would then save untold numbers of injuries or lives.
We adapt.
Some cars today already have IR sensors (night vision systems) installed in them, and include pedestrian detection monitors as well. Cars don't have to be full on self-driving to have a large increase in safety and accident prevention.
On the other hand, noone was ever run over by a mistyped URL.
Human intervention would be required to enable the vehicles to move again.
After some real-world experience, I would want to give the emergency stop system the ability to release the hold on one layer of vehicles at a time, allowing the smart AI to drive again but at a speed limited by the "dumb" system to nothing faster than human walking speed, enabling them to clear the area. If they cleared without further incident, the next layer would be released until the whole stoppage either cleared itself or was restopped because something was still wrong.
These kinds of crashes generally have the some root cause: everybody is traveling too fast for conditions, and when one thing goes wrong, the end result is that you get a chain reaction of everyone accreting into a single crash. If self-driving cars are capable of making the same misjudgements, then these kinds of crashes absolutely can happen.
Well, self-driving cars can currently handle none of the aforementioned conditions. That's why they are being tested where those conditions don't exist, and that's why we're about 20 years from widespread adoption. It rains and snows in much of the rest of the world.
Yes, I live in the rest of the world and my family has been affected by fatal collisions brought on by adverse weather. I remain skeptical that self driving cars will be able to handle all of these things. I do think they'll be okay for mundane driving where the biggest risk is driver distractions, however.
I would like to see cars and maybe even roads made smarter in order to make driving safer. If all the cars on the road were part of a wireless network and they could signal to one another when they're braking or turning, then other vehicles could respond more quickly.
Heck, these systems could even be included in phones and pedestrian crossings so that cars could be signalled instantly when a pedestrian steps into the road in order to cross.
BUT (and there's always a "but"), the problem is exponentially hard, and it can't be solved with today's technology.
I agree with you that widespread adoption will require modification of roadways, and potentially also segregation of autonomous and human-controlled traffic streams.
2) you’re describing how a system can crash. Driverless cars don’t share state, besides, I guess, vast scale astronomical and weather conditions. It’s not likely something would cause all cars to crash because of a collective hiccup. There may be a set of circumstances that consistently cause individual cars to crash, but that’s not really the same thing. And we can understand how these events are processed, so unlike people, we can adapt for future repeats of this scenario.
Errors or malicious attacks upstream could lead to disruptive or even dangerous behavior by a lot of cars at the same time.
It can be simultaneously true that we cannot describe in a causally deterministic way how a system works while also determining with some confidence that the system is performing as desired. This is true of all human systems.
So if you don't know how people drive, but have enough confidence to let them drive based on performance in evaluation scenarios, then similarly we should be able to let machines drive assuming they perform equally as well as humans over the same contexts without knowing how they drive either.
Local ones seem unlikely to cause casualties larger than events that fool large numbers of human drivers, like sinkholes or malfunctioning traffic lights or black ice on a freeway. Some of these will probably happen, and the systems will be improved, and they'll happen less and less often.
Global ones could be very large, and there are several instances of global, cascading failures in other systems like telephone or data networks. There are known ways to prevent many classes of failures, but getting to 100.000000% is hard.
For the average non-technical person, this is pretty much their relationship with all technology.
I don't think it's exclusive to tech either. I know lots of people that have absolutely no idea how the mechanical side of a car works, but they trust their lives to one at 100km/hr because it has worked generally pretty well in the past.
Car safety has made great progress in the past 10 or so years. People don't have to know how they work under the hood as long as designers and engineers already built safety into them and these designs have been heavily tested by third parties (e.g. Euro NCAP, IIHS, etc)
Self driving cars will have to prove their safety, especially in extreme weather and non paved roads.
Any system that requires a human operator to passively observe a boring process for hours at a time, yet snap to attention immediately and without warning to take control and prevent human deaths is simply insane. That's the diametric opposite of how a safe system should be designed. If the human is required for safe operation, they should just be driving at all times to maintain vigilance. I don't see people talk about that much. The assumption seems to be that AI will quickly get better and the human back-up will not be needed--that the human is kind of a temporary workaround. Well I've seen plenty of engineering projects where the temporary fix becomes a long-term pain in the ass because the difficulty of the real solution was massively underestimated and under-budgeted. We might get stuck with these terribly designed AI-human hybrid systems for many decades as AI keeps getting pushed into the horizon and we slowly realize that this is one of those hard problems that doesn't progress the same as most other tech does.
There are pure driverless systems out there, but they don't cope well with many of the incongruities and fringe situations which are an unfortunate reality on the open road in most of our country. I think they will be confined to niche uses for a long, long time.
I don't see why this would be any different for deciding whether computers should be allowed to drive - we can just make sure self-driving cars pass a road test. We could even do better than driver's license tests and have a human driver at the wheel during the test just in case.
We let people drive because we know that they are fully capable agents able to make decisions even in case of failure. In other words, they are responsible, reliable, fully cognisant individuals, far beyond the technical ability of operating a machine.
A self-driving car is like a parrot trained to speak. Not only is it not an agent aware of its larger surrounding and potential implications of getting into unknown situations. It also has no real understanding of what driving is at all, it's even stupider than a pet. And because its software is usually deployed to thousands if not millions of devices at the same time, there is little to no variance to protect from extreme failure.
I wish I could share your optimism in humanity, but what you just claimed sounds far from being true universally.
Take your typical driver and throw in a failure situation they have never trained before (because, who has time to train them for every possible, or evens few likely failures), and it is not likely to end well.
Humans have a base line of common sense reasoning and the ability to create coherent models of the world. A human driver isn't (rare exceptions aside) going to drive up an impossible street or try to dodge a street sign mistaken for a kangoroo in West Germany.
There's nice examples of how an ML algorithm can be tricked into all sorts of pathological behaviour by choosing adversarial images, there's no baseline of reason to fall back on.
Imagine a model that has one of these fatal edge cases baked in and a thousand cars suddenly mistake a flash of light for a u-turn. Sure, one or two human drivers out of a ten thousand might drive that irrationally on occasion, but compared to a machine we can usually cope with this.
And I don't even want to imagine what happens if this turns malicious. Imagine a bad actor on a bridge above a highway with some artifact that messes automated cars recognition up and the developers forgot to put some hard fail-safe in for that piece of road.
It's worth noting the downsides to this agency too, though: Agency is what makes the stereotypical male 20yo driver speed and drive dangerously to show off to his friends, which leads to a very large number of accidents.
It's unclear what accidents this agency avoids though. I can imagine a few situations where it helps (avoiding a terrorist attack by speeding away from the scene?). But in most cases the default operation for a self-driving car will be to stop as quickly and safely as possible in unusual circumstances, and it's unclear how often this is the wrong decision.
Google something like "top causes of automobile accidents". No list I found had exogenous causes (rain, snow, etc) in the top 5. For machine driving to top that isn't going to be that hard given that we are our own worst enemy on the road.
There just aren't reliable statistics on what the safety of this tech will be at this point in time. All we have is extrapolations with lots of bias and little independent review. Maybe it will eventually be worth it, but that experiment will come with a great many human deaths and we can't simply brush aside the ethics of that. I personally think it won't prove to be safer than humans for many decades. This is far more of a complex problem than a lot of the developers think, and I believe there's a case of engineering arrogance going on here as well.
On the surface it seems like we have something very close to a viable product, but the remaining little details and issues are important and represent an enormous amount of work, where we are going to see diminishing returns and enormous difficulty making a system that is flexible and robust enough to handle all the required situations more reliably than the average driver on its own for thousands of miles, scaled up to millions of units around the world. They will be good enough for niche applications, but the dream of a general self-driving infrastructure is pretty far off.
You don’t need to be better than humans for all situations. You need to be safe. Humans are generally well above that threshold, except for when they’re drunk, high, tired, distracted, or otherwise stupid. It doesn’t really make sense to compare deaths per mile for cars vs attentive humans in stable conditions. Both can honestly be reduced to the point at which they’re practically 0 unless it’s the victim’s fault. We do need to be attentive to edge cases that cause cars not to be safe, but we can also observe the overall pattern and draw conclusions on how likely unknown edge cases are to be encountered.
This happens all the time already: there is a condition in the environment (eg when it snows in Seattle) there is a systematic failure involving large numbers of people driven cars at the same time.
People just don’t have experience (erm, “training”) with the weird unusual environmental condition and fail to deal with it. Not surprising that computers would have similar problems.
> PS: Please don't insult your intelligence by replying that we don't understand how people drive either, so it makes any opaque self-driving software okay.
But we actually do know how well people drive. We have data on failure rates for sure. We don’t know how they drive, that is completely opaque.
I won't insult my intelligence; I'll ask why you trust people even though you don't know how they work and you know that at a high rate they intentionally impair themselves before driving?
Recall that time when a human trader lost a billion dollars?
https://traderhq.com/traders-cost-company-billions/
Remember when the economy tanked because everyone in finance used Black-Scholes model that seemed reliable, because it worked most of the time in the past?
But another way to look at this is the fact that "cloning" the same AI brain across multiple cars is equivalent to a single AI brain controlling many cars. And that does have human analogies where you have a single person who has a dramatic amount of power. Any flaws in that human's reasoning can be disastrous.
Trump having access to the nuclear launch codes comes to mind.
In practice, the way we mitigate this is by surrounding that human with advisors and other brains to try to help shore up deficiencies. We could do something similar with AI. Have a couple of differently-trained models and compare their outputs. If they disagree wildly, then try to gracefully shut down the system.
Unless there is a condition in the environment that causes a systemic failure involving large number of self-driving cars at the same time. This happens all the time with other types of software. People shouldn't assume that machine-learned models are magically different and ignore all common-sense rules we apply to safety-critical applications. In your daily job, would you be willing to bet your entire car on a black box piece of code that seems reliable, because it worked most of the time in the past?
Also, recall that flash crash of the stock market. Imagine something like that, but with lots of cars on a highway. If you don't understand "how a self-driving car does it", you are unlikely to install appropriate safeguards.
PS: Please don't insult your intelligence by replying that we don't understand how people drive either, so it makes any opaque self-driving software okay.