I have been thinking that too. At what point do you decide that the autopilot is making a mistake and take over? That's an almost impossible task to perform within the available time.
I can understand why this wouldn't happen from a business perspective, and it's also presumably not as simple to implement as I'm implying, but I can't think of a better way to get around the uncertainty of whether the car's operating in an expected way or not.
The "confidence meter" would almost certainly be 100% right up until it crashes into an obvious obstacle
Yeah, that's basically what I'm asking for, maybe a warning light at a certain threshold would be a better default, personally I'd still find a number more trustworthy though.
> The "confidence meter" would almost certainly be 100% right up until it crashes into an obvious obstacle
If image recognition algorithms have associated confidence levels, I'd be surprised something more complicated like road navigation was 100% certain all the time.
Unless you design the network to specifically answer "how similar is this?" and pair it with a training set and the output of another executing neural net, a "confidence meter" isn't possible.
This is one of the trickier bits to wrap one's mind around with neural networks. The systems we train for specialized tasks have no concept of 'confidence' or being wrong. There is no meta-awareness that judges how well a task is being performed in real-time outside of the training environment.
Humans don't suffer from this issue (as much) due to the mind bogglingly large and complex neural nets we have to deal with the world. Every time you set yourself to practicing a task, you are putting a previously trained neural network through further training and evolution. You can recognize when you are doing it because the process of doing so takes conscious effort. You are not 'just doing <the task>', you are 'doing <the task>, and comparing your performance against an ideation or metric of how you WANT <the task> to be performed'. This process is basically your pre-frontal cortex and occipital lobe for instance, tuning your hippocampus, sensory and motor cortex to perfect the perfect tennis swing.
When we train visual neural networks, we're talking levels of intelligence supplied by the occipital lobe , hippocampus, alone. Imagine every time you hear about a neural net that a human was lobotomized until they could perform ONLY that task with any reliability.
Kinda changes the comfort level with letting one of these take the wheel doesn't it?
Neural nets are REALLY neat. Don't get me wrong. I love them. Unfortunately, the real world is also INCREDIBLY hard to safely operate in. Many little things that humans 'just do' are the results of our own brains 'hacking' together older neural nets with a higher level one.
Calibrating the probabilities of machine learning algorithms is an old problem. By nature of disciminative algorithms and increasing model capacities, yes training typically pushes outputs to one extreme. There is a ton of information still maintained which can be properly calibrated for downstream ingestion, which anyone actually trying to integrate these into actual applications should be doing.
Most recently:
https://arxiv.org/abs/1706.04599 https://arxiv.org/abs/1802.03916
If for instance it failed to detect an obstacle how could it tell you that it failed detecting an obstacle? Or that it got the road markings wrong?
Given that the consequences for making a mistake might be people dying I certainly hope that's not how it works under the hood. Anything less than "I'm absolutely sure that I'm not currently speeding towards an obstacle" should automatically trigger a safety system that would cause the autopilot to disengage and give the control back to the driver, maybe engaging the emergency lights to let the other drivers know that something is amiss.
I wouldn't be surprised if in the moments preceding these crashes the autopilot algorithm was completely sure it was doing the right thing.
Of course these various crashes show that in fact the algorithm makes mistakes sometimes but I certainly hope that it's not aware that it's making mistakes, it just messes up while it thinks everything is going as planned.
Personally having tried Tesla AP1 iit was painfully clear I can never fully trust any assisted driving aid, they can fail at any point and I need to be ready to take over immediately. Particularly having to wrestle the steering wheel from AP1 as it slightly resists is reason enough to stop using it.
I think I'd find it a simpler metric to follow than the more qualitative information about car behaviour, as well as more useful to decide whether the car or I should be in control.
You're quite right about timing, I have no idea whether this metric would plummet quickly or gracefully drop so the driver would have time to take charge once a threshold was hit.
I want to stress that this is an underbaked idea and really just a thought experiment about what would make me personally feel more comfortable with driving a semi-autonomous vehicle.
This should be a law.
#1 job of any self driving system is to avoid obstacles. If you can’t do that reliably, it’s a fail!
It's just not good enough. Semi-autonomous cars can't be hitting stationary objects without reacting at all. If they fail at this basic task, then they shouldn't be allowed on the road.
You probably have 100ms to begin your action, 500ms to complete it.
Good luck with that if you're not perfectly alert.
The Queensland government estimates it takes drivers 1.5 seconds to react to an emergency [1] (I wouldn't be surprised if that was a 99th percentile figure, of course)
The only thing we can blame the victim for here is foolishly trusting Tesla's apparently crap software.
[1] https://www.qld.gov.au/transport/safety/road-safety/driving-...
If you have to be constantly at this level of alertness, what is even the point of having an autopilot?
Your cognitive load is lowered by Autopilot enormously. You can pay better attention to what’s going on around you and anticipate issues (say, a car that drivers weirdly in front of you) because you’re not busy with the low-level driving.
Part of that is that you see a difficult road stretch coming up and disengage the assist IN ADVANCE - becuase you’re not a complete moron, know the lane assist is for driving in highway lanes and the thing you’re approaching is decisively not one.
If you over-automate something you end up having to spend more time supervising and fixing it than if you'd just done the thing yourself in the first place.
Auto-pilot will make driving a lot more like flying is for pilots: 99% extremely dull, 1% terrifying.
A funny feeling like that, I’m ready to disengage immediately and then 1s is plenty of time. But I wouldn’t use Autopilot near an obstacle like that to begin with - anticipate, take over for the weird stretches.
Regardless, you're saying that if you're not paying attention every single second of the trip, instantly ready to rect to even the slightest deviation from your intended trajectory, then you're doing it wrong.
What, exactly, is the point of an auto-pilot system if you need to be in that state of absolute alertness?
Also unless certified on certain roads with clear markers, drivers should be required to have hands on wheels at all times.
This is a great point. With regular cars, drivers can reliably predict where the car will be after 5 seconds. However, with Tesla, there is really no way to know what the auto-driver will do, and so the driver has to stay even more attentive to ensure that it does not do something wrong. It's like driving a car that is not under your control, and may go in random directions.
Of course if users get to used to such differences then those can become 'features' which only makes progress harder.
That means the autopilot doesn't help at all since you would have to basically drive manually all the time anyway.
I would think it'd be seen as a "distraction", those suction cup mounts for example are illegal in a lot of states.
EDIT: typo
The purpose of the autopilot is to increase safety for drivers that are not attentive. However, it turns out that Tesla requires drivers to be even more attentive than regular vehicles. This is not a sign of progression.
We can differ in opinions but I strongly feel that the eventual utopia of self driving cars will be much safer than the current world. And we are making progress on this daily. There will be few unfortunate incidents but in the long term a lot more lives will be saved.
Unless you're suggesting that Tesla is justified in actively making people less safe because it helps them develop self-driving technology faster?
In that case they should issue a recall, and not use real users as their test drivers. Once they get to their eventual goal, they can resume selling autopilot cars.
> We can differ in opinions but I strongly feel that the eventual utopia of self driving cars will be much safer than the current world. And we are making progress on this daily. There will be few unfortunate incidents but in the long term a lot more lives will be saved.
I totally agree with you that in the long term self-driving cars are much safer. What I do not agree with is Tesla's approach towards achieving that goal. They are selling an unsafe product while marketing it as much safer than the existing products. This is just plain fraud.
Sorry, but this is the exact wrong conclusion to make from the available data. Partially automated cars have been driven by average people on the public roads since 2006, when the Mercedes S class got active steering and ACC. There is no data that says that these systems, which are available on cars as affordable as e.g. a VW Polo, lead to more crashes. In fact the only data I have seen suggests that they reduce crashes and driver fatigue significantly.
The issue that such a feature is systematically crashing and even killing its users is very specific to Tesla. Tesla's system has had a tendency to run into stationary objects for as long as it has been available to the public. Teslas have a significantly higher rate of causing insurance claims. There are countless videos showing it happening, the two (possibly three) fatalities so far are just the tip of the iceberg. Tesla blamed MobilEye (their previous supplier of camera-based object detection hardware), but it seems fairly evident that Tesla's system has the exact same systemic issue even with their new in-house object detection software.
My opinion as an engineer working on this exact kind of system for one of Tesla's competitors is that Tesla's system just isn't robust enough to be allowed to be used like all other manufacturer's autosteer/ACC systems. IMO that's because their basic approach (doing simple line-following on the lane markings) is super naive. A much more intricate approach needed. One that does some level of reasoning on what is a driving surface and what isn't. This has been known in the automotive industry for over a decade. Sorry to be so absolute, but frankly it's just unnacceptable that Tesla is gambling their customer's lives on such a ridiculously half-baked implementation.
> Lane following assistance without good automatic braking kills.
Correct. It also kills when it lies to the driver about how confident it is in a given situation. Many of the crashes, instances of cars veering off into oncoming lanes etc. we have seen would not have happened, if Tesla would fall back to the driver in sketchy situations. However what it does do instead is not even notfy the driver that the system is hanging on by the barest of threads. This works out fine quite often and makes it seem like the car can detect the lanes successfully even in sketchy situations. Yet once the system (and the driver) runs out of luck, there is absolutely no chance to rectify the situation.
What the GP was referring to, which is what I've see it referred to in Google presentations is level 3 automation. That is what autopilot is supposed to be, where you can take you hands off the wheel and the car can drive itself but needs constant supervision.
What you're referring to is passive assistance such as collision detection which don't drive the car but provide alerts. There is nothing wrong with this technology, it works because the driver has to drive as they normally would but it provides assistance when it spots something the driver doesn't. Here humans do the object detection and the computers stay alert as a backup.
There is a problem when the car drives itself but the driver has to stay alert. This is a problem because the human is much better at object detection and the car is much better at staying alert.
A level 2 system has to be monitored 100% of the time, because it can not be trusted to warn the driver and fall back to manual automatically in every situation. A level 3 system is robust enough to make this guarantee, which means that the driver can take his/her hands off the wheel until prompted to take back control.
Autopilot is a level 2 system, because you can not take your hands of the wheel of a Tesla and expect to not die. In fact the car itself will tell you to put your hands back on the wheel after a short while. After the fatal crashes started getting into the media, Tesla themselves have stated that it's only a level 2 system. Their CEO's weird claims of Autopilot being able to handle 90% of all driving tasks, there being a coast-to-coast Autopilot demonstration run in 2017 etc. are just marketing BS.
However I still disagree with your original comment:
> Sorry, but this is the exact wrong conclusion to make from the available data. Partially automated cars have been driven by average people on the public roads since 2006, when the Mercedes S class got active steering and ACC. There is no data that says that these systems, which are available on cars as affordable as e.g. a VW Polo, lead to more crashes. In fact the only data I have seen suggests that they reduce crashes and driver fatigue significantly.
Firstly, I don't see any examples of VW Polo claiming to have a steering assist, even the most recent 2017 edition [1] only has level 1 features [0]. Steering assist only came to the Mecedes S-Class in 2014 [2].
What I (and the parent comment you initially replied to) was tying to claim was that where as Level 1 features [0] (cruise control, automatic braking, and lane keeping) are great, level 2 (and level 3 according to Google/Waymo) is risky because humans cannot be trusted to be alert and ready to take over.
[0]: www.nhtsa.gov/staticfiles/rulemaking/pdf/Automated_Vehicles_Policy.pdf
[1]: https://en.wikipedia.org/wiki/Volkswagen_Polo#Sixth_generati...
[2]: https://www.motorauthority.com/news/1080502_2014-mercedes-be...
Tesla can easily fix this - disable autopilot if the user does not have his hands on the wheel. Disable autopilot at construction zones. Disable autopilot sooner when confidence level is lowered.
I test drove an “autopilot 2” Tesla a while back. Avoiding hitting trash cans and parked cars was much harder than it would have been if Autopilot had been off. When you’re on a road and the car decides to steer toward an obstacle, you have very little time to correct the car.
Apparently they loose the colour info and process black and white because it is easier. https://electrek.co/2017/05/16/tesla-autopilot-2-0-can-see/
This seems to have been an issue in some of the crashes and maybe was a mistake. See also for example the trailer an earlier Tesla crashed into https://www.theregister.co.uk/2017/06/20/tesla_death_crash_a...
Kind of obvious in colour but at the time it was probably a grey shape against a similar brightness grey sky
Also the big red fire truck http://autoweek.com/article/technology/ntsb-probe-autopilot-...
If that's true, it looks like a very big mistake to me, considering that we use distinguishable color a lot to signal risks.
But knowing all that, and now knowing Tesla made that short cut, there is no way on the face of this earth I am ever driving a Tesla, or for that matter, driving a car anywhere in the USA that has them on the road.
You guys have a really good train network, right?
About the US train network, it's good for freight but passenger trains are slow.
Tesla's stance to blame the driver without admitting that their hardware is inadequate will back them into a corner in the long term. They cannot transition from highway to city driving without significant changes to the autopilot software and hardware and this means they effectively have to start over from scratch. They have to start working on a serious competitor to Uber or Waymo now otherwise Tesla will be too late to the market.
If memory serves, Waymo's way seems to be summarised by the sentence "driver assist which we can have right now is inherently dangerous so let's go for full autonomy which we can't have for many years still".
A more accurate comparison of the two companies then might be that they are both trying to sell technology that they don't yet have (autonomous driving) and that only one of them is at least trying to develop (Waymo).
However, it should be absolutely clear that autonomous driving is currently only a promise and that this is true for every product being developed.
Honestly the autopilot has changed my driving experience and I hate when I have to drive cars without it.
I do agree that you need to be aware of your surroundings and ready to take over, though.
I often wonder if auto-pilot is truly 100% auto-pilot if it requires you to keep an eye out for its "Hey maybe I'll screw it up, so please keep a watch on me"
If your hands are on a steering wheel, you’re watching the road, and autopilot begins turning towards an obstacle, you should be aware enough to grip the wheel fully and prevent the incorrect turn.
End of story.
Edit: “Autopilot” is a great brand-name for the technology, but perhaps implies too much self-driving capability. Maybe “assisted steering” is a better term for this.
> End of story.
No, not true at all. If you’re in a car, paying complete attention, and holding the wheel, but keeping your arms loose enough that the car is fully controlling the steering, you somehow need to notice the car’s error and take over in something on the order of a second or perhaps much less. Keep in mind that, on many freeway ramps, there are places where you only miss an obstacle by a couple of feet when driving correctly. If the car suddenly stops following the lane correctly, you have very little time to fix it.
It seems to me that errors of the sort that Autopilot seems to make regularly are very difficult for even attentive drivers to recover from.
Is that supposed to make me feel better? If the car can go from fine to crashing into a barrier in only 6 seconds, that seems like a damnation of Autopilot more so than the driver.
It might not solve all these accidents but I think a HUD showing where the car plans to travel would work better. If you knew the car was aiming off the road before it turned, you could override the controls in time to correct. It still would require lots of attention and fast reaction time (and still may be less safe than manual driving), but it would at least be better than the situation we are in now.
If this was true, wouldn't Tesla vehicles have a higher rate of crashes, all things being equal? Is there any evidence that this is the case?
Yes.
>the Model S had higher claim frequencies, higher claim severities and higher overall losses than other large luxury cars. Under collision coverage, for example, analysts estimated that the Model S's mileage-adjusted claim frequency was 37 percent higher than the comparison group, claim severity was 64 percent higher, and overall losses were 124 percent higher. http://www.iihs.org/iihs/sr/statusreport/article/52/4/4
>>Is there any evidence that this is the case?
>Yes.
Porsche also has 3x the accident rate of Daewoo. That doesn't mean Daewoo cars are 3x as safe, it just means that people who are looking for a hot-rod buy a Porsche and not a Daewoo[1].
1. https://skeptics.stackexchange.com/questions/7605/are-bmw-ow...
This is not the claim in the linked article. The linked article claims that Tesla cars have a higher claim frequency than comparable gasoline-powered cars (i.e. large luxury cars such as a Porsche), whereas for example the Nissan Leaf has a lower claim frequency than comparable gasoline-powered cars (namely the Nissan Versa).
Put another way, if your choice is between a Tesla and a gasoline-powered large luxury car, the Tesla is more dangerous. If your choice is between the Nissan Leaf and the Nissan Versa, the Leaf is less dangerous. There was no comparison between the relative danger from a Tesla and a Leaf.
I'm pointing out that all other things aren't equal, and you can't assume from overall crash data that you can tease out statistics about how safe a specific feature of the car is.
The comparison to the Fiat 500 is relevant because while the report didn't only compare Tesla vehicles to it, that's one of the comparisons.
Is a Tesla less safe than a Fiat 500 given that it's driven by the same sorts of drivers, in similar conditions and just as carefully as a Fiat 500? Maybe, but who knows? We don't have that data, since there's an up-front selection bias when you buy a high-performance luxury car.
I wasn't able to find the raw report mentioned in this article, but here's a similar older report they've published:
http://www.iihs.org/media/2648a3bc-0c73-4036-ae00-a53a399a18...
There you can see that the claim frequency of Tesla is indeed a bit higher than all other cars they're compared to, but this doesn't hold when adjusted for claim severity or overall losses. There cars like the BMW M6 and the Audi RS7 pull ahead of Tesla by far.
So at the very least you'd have to be making the claim that even if this data somehow showed how badly performing Autopilot was, that it was mainly causing things like minor scratches, not severe damage such as crashing into a freeway divider.
Just looking at these numbers there seems, to me anyway, to be a much stronger correlation between lack of safety and whether the buyer is a rich guy undergoing a mid-life crisis than any sort of Autopilot feature.
It isn't clear whether the demographics of Tesla drivers are more reckless than that of other luxury brands or not, as you point out Porsche drivers might tend to be more interested in going fast than Daewoo drivers. For Tesla on the one hand you attract people who are interested in helping the environment who I expect to be more conscientious and maybe therefore better drivers. But on the other hand there is Ludicrous Mode.
But since Tesla could have had a lower or higher crash rate than other brands and does have a higher rate we have to update our beliefs in the direction of the car being more likely to crash by conservation of expected evidence. Unless you'd argue that a lower crash rate means that Tesla's safety features prevent lots of crashes.
An example of the inverse of this concept is the roundabout or traffic circle, which has a higher rate of much lower severity accidents than traffic lights or stop signs
- edit - Removed irrelevant edit.
Any references to support that statement?
First, how do you know it was Musk's decision? And second, when you imply that by using lower cost sensors (cameras), Tesla is cutting costs, how do you support that?
Their lower cost of sensors might be offset by having to spend more on software development for image processing, for example. When summed up, the overall cost per car may increase significantly.
At least to me, it is not clear at all whether Tesla's self-driving technology cost per car is less or more than it is e.g. for cars which use lidar sensors.
In any case, self driving cars will have accidents, not the least because some accidents are unavoidable (random behaviour of other cars, animals or pedestrians, unexpected slippery roads, etc), but running heads on into a large visible static object should never happen. It’s a bug.
you are the one telling anecdotal experience, jfyi
Here's a good example - a Tesla on autopilot crashing into a temporary road barrier which required a lane change.[1] This is a view from the dashcam of the vehicle behind the Tesla. At 00:21, things look normal. At 00:22, the Tesla should just be starting to turn to follow the lane and avoid the barrier, but it isn't. By 00:23, it's hit the wall. By the time the driver could have detected that failure, it was too late.
Big, solid, obvious orange obstacle. Freeway on a clear day. Tesla's system didn't detect it. By the time it was clear that the driver needed to take over, it was too late. This is why, as the head of Google's self driving effort once said, partial self driving "assistance" is inherently unsafe. Lane following assistance without good automatic braking kills.
This is the Tesla self-crashing car in action. Tesla fails at the basic task of self-driving - not hitting obstacles. If it doesn't look like the rear end of a car or truck, it gets hit. So far, one street sweeper, one fire truck, one disabled car, one crossing tractor trailer, and two freeway barriers have been hit. Those are the ones that got press attention. There are probably more incidents.
Automatic driving the Waymo way seems to be working. Automatic driving the Tesla way leaves a trail of blood and death. That is not an accident. It follows directly from Musk's decision to cut costs by trying to do the job with inadequate sensors and processing.
[1] https://www.youtube.com/watch?v=-2ml6sjk_8c