There's a term for it: "normalization of deviance"[1]. A highly risky thing is done and doesn't fail as early as thought. The people doing it assume that means the risk is lower than they thought, but it's not, and they just happened to get lucky.
I'm sure Elon knows this, so probably believes their tech can advance enough before their luck turns. I personally think he's deadly wrong on this, but it seems like a lot of Silicon Valley Tesla owners who sleep or read their phone while on Autopilot accept it at face value.
[1] - https://en.wikipedia.org/wiki/Normalization_of_deviance
I believe this is not entirely correct, as the parent talks about how a company can get away with something, while the term you use applies to the company itself.
The main question should be: why do we (the government) even allow companies to do something like this?
Legal : If you sell it as a drive assist and not a driver replacement, you're off the hook, with a high probability. It's on the drivers if they get complacent. There may even be precedents with similar products for trucks.
Moral: pushing this now will very likely kill some people that would otherwise be alive (drivers or pedestrians). But it's very unlikely to be a carnage - I'd dare say even impossible. On the other hand, moving the technology forward and directly into widespread commercial use will save people, and likely even... what's the equivalent word for a "carnage" but where people are saved? Miracle.
Branding: well, there you have a real risk. Public opinion is fickle, and a particularly gruesome accident might do some real damage, both to Tesla and to self driving technologies in general. But then, they do have a high risk high reward mentality, and the chance is good that the first bad accident (statistically inevitable) will come after the advantages are proven and well established.
This is the only rationale I can come up with for Tesla's approach. The tenets go like this:
1) The ultimate realization of self driving will be successful, and will have profoundly positive impacts on society (e.g. safety, urban planning, ease and availability of transportation, economic)
2) The path to that ultimate realization contains many local optima traps
3) The only way to push past those local optima is by acquiring large amount of real world automated driving data, using systems in production
From a technical standpoint, I'm disinclined to disagree with them on the necessity of large amounts of real world system behavior data as a key ingredient.
Personally, I also think we (humans) have become way too culturally risk adverse. Or, to put the calculus another way: if I have a credible belief that I can save some number of future humans, what is the acceptable moral number of current humans to place at risk and/or kill?
Imho, that's a grave, serious decision, but not one to which the morally absolute is "zero, always."
From a purely ethical standpoint (leaving legal aside), the crux would seem to be informed users. Not all Tesla drivers are likely capable of understanding the current SotA, and I'd hope (and expect) that Tesla's phased rollout of the beta is informed by previous user interaction with assisted driving.
> Imho, that's a grave, serious decision, but not one to which the morally absolute is "zero, always."
I think that what we've learned throughout human history is that no one should be trusted and given the power to make such decisions. It has invariably led to disasters. The very idea that someone can decide to put someone else's life in danger for potential future advances gives too many perverse incentives - chief of which, rationalization that "more people will be saved" even as the experiment crashes and burns.
When it's for the greater good, or by the winners, we conveniently omit it from history.
When it's for the greater evil, or by the losers, we publicize it as a cautionary tale of moral hubris.
Example 1: Vaccines
This is an interesting example because the winners and losers seem to be proportionately spread across almost all of society (except anti-Vaxxers who benefit from herd immunity but don't sacrifice with the minute risk of vaccines). See also: https://www.hrsa.gov/vaccine-compensation/index.html
Example 2: War on Crime / Safety
Another example would be the war on crime. Liberties and lives are sacrificed in the name of safety for others. Unfortunately, the liberty/lives sacrificed fall disproportionately into one group, and those protected fall disproportionately into a different group group. If you look at the 2nd order effects (impoverishment, single-parent households, effects on education and future) lots more lives are lost, again in a subset of society.
This is clearly possible in principle. After all, fully autonomous vehicles can have much more capable sensor systems than any human driver, can process the input from those sensors faster, and can then apply any necessary adjustments to the vehicle's behaviour almost instantaneously, all without ever getting tired or distracted. No human driver will ever match that potential.
The problem I have with this whole argument, and thus the reason I'm not sure the moral/ethical arguments stand up either, is that we don't know how to reliably write excellent software yet.
The entire argument for autonomous driving rests on the premise that we will at some point in the usefully near future reach a situation where the cars drive themselves much better than we humans collectively do. But there is, to the best of my knowledge, no evidence so far that we as a society can actually create the necessary software yet.
Until we can, any autonomous vehicle control system will be at risk of life-threatening bugs. In particular, it will be at risk of widespread failures of the same type that could cause much more harm than any single human driver's failure could, whether caused by defects in the software alone or by a malicious actor exploiting a security vulnerability. That is a huge risk that is getting far too little attention so far IMHO.
The first is whether we expect the autonomous vehicles to become safer than the population of human drivers they would be replacing under normal circumstances. This is a population measure, and it's about overall safety and averages, and given enough time and data it seems a realistic outcome.
The second is the risk of a single catastrophic failure, which is a risk that doesn't generally exist with today's human-controlled vehicles. It is no exaggeration to say that the kinds of autonomous vehicle technologies we are conceiving here have the potential, if they go wrong, to cause more harm than any WMD ever deployed in war. And not only the theoretical potential, but given the current state of our software development abilities, a non-trivial chance of it actually happening in practice.
Until we've dealt with the latter problem, which I don't believe we currently know how to solve, any arguments based on the long-term superiority of autonomous vehicles over the capabilities of any human driver is going to be fundamentally flawed.
This is what it really comes down to.
Imagine if we invented the car today in our current cultural climate. I don’t think it would be possible for humanity to reap it’s advantages.
Back when it was invented people actually had to drive the things around on public roads that had a bunch of pedestrians walking haphazardly accross them continuously.
I was an autopilot skeptic, and it’s entirely for this reason. I didn’t think it would be possible for anyone to actually do what needed to be done to get this technology across the line. That is to say, give it to customers before it is safe.
If Tesla is actually able to keep doing this without it getting killed politically, then I think it actually will happen, and that gives me some hope for humanity more generally.
That, I don’t see ever being true. What would you get from that that you couldn’t get from “large amount of real world human driving data, using systems in production”?
Or to put it another way, every time the driver's inputs diverge from the automated system's inputs, the test run is invalidated. You're no longer getting results of what the automated system did.
In Germany they lost a case about their Autopilot advertisement. So not even legally they are in the clear.
They aren't just using their own customers, they are using all of us.
The cars won't be just crashing into inanimate objects, but school buses and ambulances. The QA is being done to all of us and our loved ones near their cars.
I'm having a hard time parsing your double-negative, but when shadow mode is running, it's choices are back-tested in real-time with the driver's choices, so I'm not sure how this fails to replicate what a real-driver would have done in such scenarios.
e.g. You see the driver swerve for no apparent reason, did the driver simply drive more recklessly than the car would, or did they avoid an obstacle the model didn’t catch?
People get easily distracted, get excited, get tired, all of which are propitious to bad decision-making.
Many human drivers are downright dangerous, but that's not a reason to add dangerous machine drivers on the road, either.
https://m.youtube.com/watch?v=xj73Q3lWvFM
It's been a problem for years now.
And FSD isn’t orders of magnitude safer. They are barely as good as human drivers and only in some conditions.
For the worst situations they aren’t comparable at all (since they won’t drive at all)
This must be repeated over and over for some reason: people will never accept self driving cars that are just as good or bad as human drivers. Nor should we. It’s a much too low bar.
Self driving cars will need to be orders of magnitude safer than human drivers to be even remotely acceptable.
I hate almost everything about Tesla and their business model but the idea of full automatic driving, once vetted to even a bare minimum level of approximate human parity, is enough to make me consider one anyway.
You are in between two walls and there are two straight lanes. Lane 1 has Tesla FSD coming in at full speed. Lane 2 has a human driver coming in at full speed. You can choose to be in one lane. The FSD and the human can both see you only 50 mts out and if they brake as soon as they see you, you will survive.
Which lane will you choose?
Also, there is no moving fast and breaking things here in real terms. Drivers are and have to pay attention while driving, and Tesla is strictly monitoring this with beta. You also specifically have to opt-in to beta software. 99% of Tesla owners do not have this functionality.
(I think you're probably right they don't, HN international and diverse enough, but I wouldn't be astonished.)
Note that it doesn’t mean I love it; I just expect that they will figure it out.
Tech demos have a way of setting good impressions that are not realistic. I have a feeling that's exactly what is going on right now, but a few more close calls with crashes and the shine wears off.
Previous AI systems generally fell over when they hit the final 2% of real world scenarios, unable to cope with the explosion of complexity.
In contrast, modern AI finally provide a toolkit for "last mile AI" (proverbially speaking), by allowing incorporation of those scenarios without breaking the entire previous model.
And the barrier to surpassing that problem has been hardware (now approaching solving) and calendar- & person-time.
The problem with 1970s AI (ingnoring hardware) isn't that we couldn't build useful AI, but that we couldn't afford to build useful AI in most domains.
Calendar time costs money. People time costs money.
Deep learning and derivates finally provide a solution where we can substitute machine time and data volume for calendar and people time. As a result, we can afford to do things in domains we couldn't before.
Citation needed. Deep learning is used to power some of the biggest money making machines in the world (like google or facebook ads) and they're still bad. It's also used to power products like voice recognition and it'll still add dog to my shopping list instead of soap.
Where I'd expect to see interesting developments (hopefully published openly) is in sentiment analysis / disinformation identification, as companies begin to see it as an existential threat to the unregulated nature of their businesses and fund accordingly.
One should do sentiment analysis on these discussions threads over the years to see how the attitudes have changed.
A much less dangerous version of this was the VR hype train
As you point out, all it will take is a few fatalities for public opinion to harden against self-driving, and laws will start banning or severely restricting it. Rather than being wowed by Tesla's gung-ho approach, boosters of self-driving technology should be angered by it.
The other stats in this thread make it sound like Tesla is not even the same league as other attempts at driving.
And yes, human drivers might be statistically worse - but just because this might be statistically better(and I'm not even sure that's actually the case here) it doesn't mean it should be allowed.
I'm not saying Tesla is making a good move here or maybe isn't being careful enough. I don't know yet & am not familiar with how the engineers feel about the development of the product & how bad the corners they cut during development are. That being said, the standard you've laid out is not a good world to live in. I don't know if the tech equipped in the car is sufficient.
A huge statistical difference in quality should be sufficient to go with automated cars even if they don't make the same mistakes as humans or sometimes make mistakes humans don't. "Statistical difference" isn't about a coarse level analysis either but situational. If they get into fewer road accidents but kill pedestrians at a larger rate, that's a problem. The problem is the news media only reports the top-line number because analysis is hard.
Most of this automation operates under certain assumptions about its environment and in some cases shuts itself down in a controlled fashion to fail safe in ways that may not be available to cars on roads, but these are not new ethical dilemmas.
But to pick just one example, automated saw stops and controllers aren't 100% effective at preventing injuries, but they're still a huge safety improvement over the human population at large just winging it.
Nothing is 100% safe. By your logic we’d have no medicine, have you ever read the side effects of some medicines.
You picked the worst example possible.
Medicine is one of the most tightly regulated fields in the world with huge upfront cost and approval processes spanning years or even decades. E.g. the research and approval process of a new antibiotic costs multiple billion dollars before it can be sold.
What you see now wrt the Covid-19 vaccine is an international effort that is fast-tracking everything. Regulators are willing to overlook a lot for it to come to market as fast as possible and it's still taking a year or more before it's approved for use on everybody.
Uncertainty – there is never 100% certainty when determining reasonable assurance of safety and effectiveness of a device. However, the degree of certainty of the benefits and risks of a device is a factor we consider when making benefit-risk determinations.
And regulation in no way - in no way - addresses the fact that medicine isn't close to 100% safe before adoption, and neither should self-driving cars be. As long as they are safer than humans, that's enough. Any other conclusion will cost lives.
I'm having a hard time understanding your position as to why the technology should not be allowed, even if it were to perform statistically better than humans. Is this more of a philosophical perspective?
Why wouldn't a community of engineers welcome a statistical improvement over meat drivers?
It's a lot like the "old space" approach to spacecraft engineering, spending decades and hundreds of billions of dollars to develop a rocket that would've been cutting edge half a century ago, all in pursuit of making the initial product perfect.
This would be a much more compelling argument if the technology in question weren't replacing an existing system that is frequently lethal. I do agree that it needs more testing and possibly regulation.
Then no human should be driving on the roads either or at least every vehicle should have a breathalyzer in it to check your blood alcohol level before the engine can turn on. You're being unreasonable.
Either way, it’s definitely new territory for us. I am sure there will be debates on this forever but I foresee reasonable arguments by both sides.
Heck, modern cars are full of things that control aspects of driving for you: Cruise control. Automatic emergency braking. Brake assist. Antilock brakes. Traction control. Many of these features are so beneficial to safety that they are mandatory for new cars.
If we could achieve cycling usage at the level of the Netherlands, mass transit usage on the level of Japan, and road safety on the level of Norway then we would cut road deaths by 90%. These are ambitious goals but can actually happen and are not an open-ended research project.
can actually happen *if you live in an urban area
What's missing, using this analogy, is that it is only a net loss to slow down the vaccine that actually works, which we don't know a priori. But if you fast track something that kills a bunch of people, then it might have been a waste. You might not learn enough for it to be worth the cost.
What if we push out Tesla's cars and all we learn is that vision based neural networks with the particular models tesla's using today aren't good enough? How many lives would you sacrifice to learn that when a big investment in a more careful testing strategy might say the same thing?
To say nothing of the criticisms of the utilitarian perspective. If we believed in it as a society, we'd round up the poor/unskilled/uneducated for mandatory medical experimentation, but thankfully that idea is horrifying.
I too am shocked that anyone is ok with this, given the auto industry's history of fighting against safety and transparency.
Note: I’m not going to say Tesla isn’t being at least moderately reckless with its rollout of self-driving features, just that I firmly believe even a less than ideal AI with good enough training and sensor data can trounce the safety record of human drivers.
This is the crux of the issue. If we had AI that could outperform human drivers and make our roads safer then I'd agree that we should use it. But I see no indication that this is the case, or that it's even technologically possible at this point.
Tesla can make whatever claims they want, but until their self driving systems have been independently and rigorously tested they shouldn't be anywhere near public roads. Have such tests happened yet? The article doesn't mention any, so I assume that they haven't.
Regulate and validate the vehicles and their software updates, not the engineers.
Edit: Or get used to our field moving as slowly as the bridge building field.
Holding people responsible doesn’t decrease or slow down innovation. It alters the incentives to align the innovation with public benefit. If properly caring for society is too slow for you, your part of the problem.
Oh did they?
https://en.wikipedia.org/wiki/List_of_bridge_failures#2000%E...
Edit:
> Holding people responsible doesn’t decrease or slow down innovation.
Citation required. This is strongly non-intuitive.
First one only sagged, no catastrophic failure, no deaths, no injuries.
A few more on that list were over a hundred years old. Not sure when he started holding engineers responsible.
Then you have the bridges rammed by ships, terrorism, construction accidents, a wooden suspension bridge in the middle of nowhere that had an upper limit of three and was used by seventy, etc. .
Seems as if the bridges themselves have held up fairly well.
It doesn't prevent you from trying to introduce new technology as long as risk is acceptable. Otherwise we'd still ride horse carriages (with special horses bred to be slow).
Stuff like that is only possible in the US. Would be unthinkable in the EU. Maybe in UJK but you know, they are no longer ...
1. https://europe.autonews.com/automakers/audi-bmw-others-frust...
This is analogous to banning airplanes until they can be proven safe to fly in. The countries that don’t ban them will lose more lives early on, but they’ll have a technological advantage. The place where this analogy breaks down is that hundreds of people are killed by human driven cars every day. Every year cars kill 120,000 people in the US & Europe. If we can develop and deploy this technology a year sooner, we can save tens of thousands of lives.
Waymo is at least as safe as the average human driver, while Tesla has unleashed literally worse-than-drunk drivers upon us all. Heaven help us.
As a concrete example: if you assume that 99% of accidents that would be caused by the system will be prevented by driver interventions, and you think overall the system will reduce the likelihood of an accident by 20%, the question you have to ask yourself is: if 1 in 5 people harmed over the next 6 months we could have saved, should we avoid doing so and wait until only 1 in 200 will not be harmed, in part due to their own negligence? As you can see, the assumed probabilities matter a lot, so I'd be curious how one can come up with good projections. The only case where these probabilities don't matter is if you believe that engineers ought to never create harm where none would have existed otherwise, but when it comes to self driving cars, this is an impossibility since it assumes perfect autonomy.
In general, if you buy into the theory that "human + AI" is always going to be smarter than just AI, then arguably even in a world with great autonomous driving, the best system will be one that has both an AI and a human. And that is what we have here, so it's more a question of if AI quality is sufficient or if this is just a bad idea in general given the natural tendencies of humans to be poor co-pilots.
They already claim that drivers must pay attention to the road in order to prevent accidents and this has been the status-quo in the automotive world for a century or so. Selling products that can kill if misused is not remarkable.
This could turn ugly if it's shown that enabling self-driving results in much more accidents but there is no data to support this.
if your moral premise is saving lives, your conclusion does not follow. vehicles are a leading cause of death and injury, so this omelette is worth breaking a few eggs, no?
AV has the potential to save a lot of lives. of course, there will be effects in other markets like insurance, new car sales, road repair, junk/salvage yards, tow trucks, etc. what the net economic effect will be is up for debate.
Waymo and Cruise do it this way, and their software gets better without killing anyone. Uber and Tesla figured they'd play it fast and loose and killed people.
It's not certain that Waymo's approach will be better, especially at scale (and in cities other than Phoenix). We don't have that kind of data yet. And even if it is better, Tesla's approach is still a pareto improvement over now. The perfect is the enemy of the good.
If they really thought some eggs need to be broken to get _their_(because they are using the data nobody else) systems up to speed, they could just come forward and say it and more importantly pay for the consequences.
All car manufacturers sell products that can kill people.
Yes, that's exactly why most of them have much better QA than Tesla.
And where do the other people on the road factor into the decisions of you and your 5000+ lb vehicle?
Let's start "Falsehoods self-driving programmers believe about roads:"
All highway exits are to the right from the right-hand lane. If the left lane line curves away, the road is curving and you should follow it.
You realize that those two statements are in conflict with each other right? If they're misleading and favoring Tesla then you need contrary stats to state that. However you state that they're not available, so your previous statement can only be a lie.
Any way you cut it, Autopilot and FSD are statistically worse than human drivers.
This is the only data Tesla has released comparing similar cohorts of vehicles (i.e., modern luxury cars). The only stats in which Tesla beats the average driver is where they include all cars ever put on the road, including semi trucks...but under those models every other modern car is also better than average.
[EDIT: And this is the Forbes article on Tesla's report. There's also a Cleantechnica post on the same report as well. https://www.forbes.com/sites/bradtempleton/2020/10/28/new-te...]
If Tesla FSD was 3x safer than human driving Musk would be bragging about it. The fact that he's not indicates that my interpretation is the correct one.
does it even become profitable before the 25 year mark in the current market?
(1) None of the companies involved have ever had the motto "move fast & break things." Zuckerberg created that philosophy to Facebook. It's foolish to apply it an entire industry.
(2) They are no more "using its customers as QA" than pharmaceutical companies do when gradually ramping up trials that also kill people. There has been heavy testing before this point, and it seems entirely reasonable to scale up testing before a general release.
If Tesla is going to be a safer driver than me, it’s already a good investment. If a software bug means I die, well, human error might too.
Also note the double speak of Musk on the topic.. he was the first to talk (even boast) about how the structural design made the car the safest of all .. but now safety is not as essential, it's ok to have some accidents. And it will improve their machine learning data set right ! /s
I’ve had that happen a few times in the decades I’ve been driving. It appears to be a regular occurrence in this software.
I don't understand this point of view. Cars WILL kill people with AI or humans driving. AI is not great right now, but it's already way better than humans in several situations.
Then people get spooked by bugs that the AI brings because they're not "normal" accidents, and think it's a bad idea to continue to train the AIs.
But why? Why is a "beta" AI killing people worse than the normal everyday drunk/distracted/bored/texting human driver?
Every stat that I've seen that comes out with the machines better than the people isn't a like-to-like comparison. Every like-to-like comparison I've seen has the machines still worse than people, so we don't even need to have that argument. Beware the propaganda.
If you take a shitty driver and compare it to an AI, I doubt the AI will perform worse when it comes to preventing accidents.
Sure, a skilled human driver will likely perform better than this early stage AI, but if even 1% of the bad drivers out there switch to an AI, that's a net positive in my view. Not to mention all the other risky situations involving alcohol, sleep deprivation, etc. where an automated response can save lives.
You didn't address my main point though. Why is a driverless accident worse than a drunk driver?
I really don't understand how a company can get away with using its customers as QA when they sell products that can kill people. It's one thing if it's a website for throwing virtual sheep at another; nobody really cares if your Facebook feed omits an update or two. It's another thing if a software bug means that you die. Or at least it should be - I'm amazed at the number of people who seemingly don't care.
I also wonder what this is doing to Tesla's brand reputation. I was just in the market for solar panels and Tesla's offering (both the panels and the solar roof) was very, very attractive on price & aesthetics, but their reliability record with their cars made me think twice about buying a 25+ year product that keeps my home powered and dry.