I saw a number of Tesla vehicles on the road as well, each passing these same emergency vehicles, but not always in the same lane.
By this point, there must be 1M+ incidents of Tesla vehicles passing roadside emergency vehicles in an adjacent lane.
12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases.
IMO, it is good that there is federal investigation into this. My main concern is that is seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things.
How do you know all of them were on autopilot?
>12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases.
It's also 12 more than any normal attentive human has gotten into, and an average attentive human driver is the absolute floor for self-driving on public roads unless Tesla is going to go to the lengths GM has to ensure the person behind the wheel is remaining attentive.
>IMO, it is good that there is federal investigation into this. My main concern is that is seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things.
I think they're good at understanding if Tesla loses complete public confidence in self driving that is a net-loss for humanity and technology, as well as a prime source of political shitstorm when people ask why something wasn't done sooner.
Can you clarify this please? Are you asserting that prior to the 12 Tesla incidents, there weren't any crashes into stationary emergency vehicles?
Even if the accidents a Tesla does get into are ones that a normal attentive driver wouldn't get into, if it avoids just as many accidents that a normal attentive driver would get into isn't that enough?
A lot of people probably feel the same.
But... no one is forcing anyone to use this system. They elect to. Thus this is the wrong question, but rather if drivers should be given the option to use the system.
If anything it might be reasonable to infer that poor drivers are more likely to use this system, so the standard could plausibly be below that of an average driver while still improving overall safety.
>I'm not going to trust a system that isn't as good as a normal attentive driver.
As I understand it, we're not expected to trust the current incarnation of Tesla Autopilot. We're supposed to be ready to intervene in case it screws up.
I'd be interested to know how long the drivers of the 12 Teslas in these accidents had to intervene but didn't (how long was it obvious Autopilot had screwed up before the impact).
So if data show that people keep failing at it we have to take a look and decide that it's not a good system, not that the people are at fault.
It’s not just autopilot.
I don't, but statistically speaking, some of them likely were.
Tesla has shipped about 1.6M units globally, with most of those in the US. Let conservatively estimate half of those stayed in the US, and half of that half have used autopilot frequently enough for a high degree of opportunity of these kinds of crashes. That is still 250,000 vehicles. 12 of those being involved in crashes is a pretty low number, and we also do not know how many scenarios they avoided that a human might have missed.
> It's also 12 more than any normal attentive human has gotten into
What is your definition of "normal attentive human"? There are tons of studies showing how humans suck at long-term tasks that are repetitive or non-engaging (like highway driving). People get distracted all the time while driving and cause accidents. Sure an "attentive" human would not run into a police vehicle, but there are not enough of those on the road most days.
Are you comparing a fleet of cars driving millions of miles to the performance of one ostensibly good driver? Ok, I guess we’ll need perfect systems, see you after the heat death of the universe.
It's about the unknown unknowns. If you have Teslas slamming into stopped emergency vehicles with their lights on, and you don't know why, what else don't we know about their ability to perceive and reason about the world that they're navigating in?
That said, I suspect there is some systematic issue with Tesla's system and flashing lights, and I bet they are working like demons to identify and fix it already.
Humans do a lot of dumb, irresponsible things on the road, but this is not one of them.
This cop[1] tells a lady not to stand between the vehicles because she will get killed if his patrol car gets hit, and seconds later it happens and he says "just like that".
>this is not one of them.
Yes[2], this[3] is[4] one[5] of[6] them[7].
[1] https://www.youtube.com/watch?v=fZDIwaRWCl0
[2] https://www.youtube.com/watch?v=97D3llj0KSQ
[3] https://www.youtube.com/watch?v=pV5FrWTATDY
[4] https://www.youtube.com/watch?v=loex5U9zD0s
[5] https://www.youtube.com/watch?v=khz9hlpvhko
Just like anything else, it may not be common (or it may be 'uncommon') but, to say that it never happens, is plainly, clearly and objectively false.
The authorities are simply viewing Tesla's credibility as undermined. They ask themselves, if it can't do this, how can it get anything right, or there must be many more flaws.
The humans do not like to pass over control to machines unless the machines do it clearly better.
And in this case, the authority does not want to have their credibility shot by looking away from the Tesla issue, it cannot be pushed under the metaphorical carpet anymore, there is a bulge under the carpet visible to anyone who wants to see.
> seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things.
Not sure how you think it's good then.
> 12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases.
That perspective is little more than spin. Most serious flaws are like that.
Any failure that isn't a "tiny tiny percentage of the total number of cases" is pretty serious. If it was even a small percentage (say 0.01%), Tesla autopilot would be banned from the road, and maybe Elon Musk would go to jail (or at least Tesla would probably get bankrupted) for releasing such a hazardous product.
I have been using autopilot in Nissan. If the car were to crash into an ambulance light going off I would not be able to predict it and stop it from happening quickly enough. I just have a whole lot of trust in some basic things the car does.
And this is one of those cases where they should have been able to get this right after the first crash but they didn’t and now the feds have to get involved. A lesson on why we can’t have nice things.
> 12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest.
It's really surprising what people will put up with. IIRC, recently "autopilot" had some regressions in the situations it could handle, and Elon Musk tweeted something to the effect of "Yeah, this version sucks since we're retraining all our NNs."
Like how did something that half-baked ever make it to production in a safety critical system?
I'm sorry, but how can you form an opinion so quickly? 12 crashes out of what? Any number? 12 crashes out of 10000000 scenarios sound like a pretty good record, while 12 crashes out of 15 scenarios indeed doesn't sound like a great success.
But what information do you have that the rest of us doesn't?
It doesn't really matter. They're stationary objects; there's absolutely no reason, good or bad, for the algorithm to determine that the correct trajectory change is into the stationary object.
Humans crash into stationary objects because they're lacking attention almost 100% of the time, whether it's because they're half asleep, drunk, texting, or distracted by just about anything else. Autopilot shouldn't have this issue, so there's no reason for it to fail to recognize the obstacles unless it's been poorly trained, and there's absolutely no reason for it to recognize an obstacle as a valid path or destination.
Sure some AI gurus will claim that humans can detect this objects only with 2 eyes but they forget to mention that behind those 2 eyes is a brain an actual intelligence that is aware of the world and it's place in the world. So if you can't create an actual intelligence then at least put the effort in feeding correct data into your AI and remove the line of code "if static object ignore because too hard to detect if is a truck or a shadow/paper bag/sign"
A self driving system needs to have a 3d understanding of it's environment. It's not acceptable to crash into paintings by Wile E. Coyote.
Tesla should also be prohibited from pushing updates. When the software is updated, the code should be tested for at least X thousand miles and X months before a new version may be pushed. Letting an authority do the pushing avoids the temptation of manufacturers to push updates prematurely. It also solves the problem of unexpected changes for the user at random times.
If safety issues are detected, the automobile (or at least the autopilot) should be disabled until the code has been fixed and fully tested.
What systems are you referring to? Lidar and radar systems can't reliably detect stationary objects as hazards, because most of the world is stationary and consequently gets filtered out.
The lidar/radar detects the objects the AI should label them and decide, if it can't do it then throw more video cards at it and more training or find some competent people that could use more deterministic code to reduce the complexity before the AI attempts to run.
https://www.wired.com/story/tesla-autopilot-why-crash-radar/
How a code that decides what to ignore or not is a sensor issue? Sensors tell you that there is something in front, sure maybe is a sudden thing and there is no velocity/acceleration vector for a few moments but the object is detected ...the software DECIDES what to ignore.
The example where a car changes slane and the code ignores any static object in front of that car seems a clear case of a crash waiting to happen, the car in front might change the lane to avoid something. (and the fact that some other tech that is honestly labeled lane assist has this issue does not excuse a self claimed autopilot to have same flaws."
Let me summarize and let me know where I am wrong:
"Because the software is bad and it triggers too many false positives the coders added a "clever" hack , we ignore most static objects."
I guess "Sensor Module" would be a more accurate description.
Most automotive systems are designed as a series of modules that function independently but jointly form a system. They either broadcast or consume messages over the CAN BUS.
The Volvo V60 for example has a Forward Sensor Module, the radar, which outputs a handful parameters like Primary Target Range, Voltage, and Automatic Alignment Offset. It's the responsibility of other body modules to consume that data. The Closing Velocity Module might then broadcast Acceleration related messages which intern would be picked up by the Adaptive Cruise Control Module. Or it might broadcast Collision Warning Messages which the Stereo Module reads and responds by blaring a warning noise out the speakers and also the Brake Control Module receives and applies the brakes.
Though the Google car seems to have strong enough hardware to keep track of many vehicles and objects at a big enough distance to not be surprised if something appears or disappears for a frame.
Not sure what people want for a sensor, to detect and label the objects for you? or calculate velocities and predict the future? That seems to be the software developers job but you might need competent people and not throwing big data into a black box and retrain until it mostly works.
It would be interesting to see a breakdown of the 12 incidents and the Tesla Models involved. Were the incidents involving radar/vision blended, or vision only vehicles or both?
If the incidents only involved older vehicles that were radar equipped, that would be good PR for the vision only system. If the incidents were in vision only systems, it would be bad. I suspect it'll probably be a little of both.
Most systems are self contained modules with low power micro-controllers that can emit basic warning and alert data onto the CAN BUS. Tesla's is actually taking the input from it's sensors and attempting to do more in-depth analysis and actually produce a 3D space.
The fact that they're still suffering from the same problem as other manufacturers shouldn't be an admonishment of their technology or attempts unless or until it's proven that their approach is fundamentally worse.
If their approach fundamentally means they decided to work around mapping their surroundings in 3D correctly, as the GP suggested, there is a strong reason to admonish.
A autonomous driving system which can't detect stationary obstacles is not an autonomous driving system: It's a murder machine.
If they can't do this, they shouldn't operate, full stop.
There's no reason to kill autonomous driving just because gen 1 - 50 isn't perfect but is an improvement on current driving methodology.
Giving a one-line sentence of what happened disregards other variables such as, lighting conditions, where the stationary object was (Over a hill, round a sharp bend, etc.), how much stopping distance was there. A lot of Tesla hate is purposefully done from EV haters & Musk haters. No one can argue that there isn't an establishment who wants to tear down and prevent Tesla & Musk from succeeding. Teslas will catch on fire (like other cars do), Teslas will crash (like other cars do), Auto-pilot will fail (while also being overall an improvement on driving safety).
We can't assume the stationary object was 300 feet away on a straightaway in a perfect visibility conditions, we have to judge whether a human would've performed worse and whether the same cases are occurring over and over. 12 crashes over a few months is a pretty tiny fraction when there are probably more than 12 humans crashing into stationary objects every hour.
>> The horrible truth is that if autonomous driving can save 1+ life, it's better than what we have now so we should keep it and accept that crashes will occur >>
As that '1 life' saved may be countered by a number of lives lost due to the system that might otherwise have survived.
This argument ignores the reality of driving: by being a competent, attentive, careful driver you can avoid many classes of accident. Crashes are all about fault, and as a safe driver you can avoid most at-fault accidents. You get to avoid all the self-inflicted damage of drunk/incapacitated driving, unsafe driving for weather conditions, extreme speeding / reckless driving, etc. You still have to deal with the externalities of those, but some fraction of the time you can notice and anticipate these people on the road and stay away. Not always of course, so that still leaves some number of deaths per mile, but I'd bet that threshold is way lower than the "save one life" bar. So if you're a good driver, autonomous cars at the "current death rate" safety bar would make you less safe.
What you get by ignoring this and using the current death rate as a bar is a "transfer of deaths": you take all the deaths that were the fault of the driver (driver dies due to being drunk/high, very reckless driving, etc) and transfer them to random innocent people who were not at fault.
It’s funny you use the word reality since this is completely at odds with driving in America. You’re comparing an idealized world of competent drivers to the real world where there are basically no barriers to getting a license and everyone is on their phone behind the wheel. Unless you advocate for stricter licensing and more police enforcement against inattentive driving with equal vigor, this line of reasoning doesn’t hold.
So I take it you would be fine with the nightmare version of Apple's phone scanner, continuously scanning your devices for thoughtcrime? After all, "If it saves just one child..."
Any time I hear emotional appeals that ignore the cost side of the equation, I look for a financial motive.
The problem with the Apple system is significant downsides that are completely separate from CSAM.
This is not true at all and we have much higher standards than this. This is like saying if a vaccine cures Covid but manages to kill people at a rate slightly less than Covid, we should embrace it.
Are you saying it doesn't matter how often it happens? I'd be a lot more worried about things if something happens always compared to it happening in some cases. Sure, stationary objects should be driven into, I agree. But if this only happens 1 time over the course of 10 years, I'd be a lot more lenient than if it happens once per week, no matter if the reason is "good or bad" (not sure what that means?)
You claim that you have some super duper AI but it can't detect a truck or a wall from a shadow or bag , then your AI is stupid and buy some better sensors and improve your AI.
Well you don't ask for much
https://arstechnica.com/cars/2018/06/why-emergency-braking-s...
(thanks for the link correction)
Instead of that, we got robots that are apparently unable to avoid crashing into large vehicles with flashing lights. Sometimes.
As usual, the future never ceases to disappoint me.
I believe you're thinking of terminators, not automated drivers:
https://quotegeek.com/quotes-from-movies/the-terminator/1136...
Though the "will not stop, ever...", even for an emergency vehicle, might be relevant.
These are situations where failure is not suppose to happen when the driver is paying attention. Given that the auto-pilot is always paying attention when operational, this becomes an unacceptable failure mode because it failed in the most rudimentary way - failure to perceive stationary obstacles and a display of complete lack of situational awareness.
For emergency responders on highways, there are means to reduce the risk of additional collisions by increasing visibility to oncoming drivers (markers, lights, etc.). If the auto-pilot is not able to perceive these even when paying attention, we do have a problem.
I guess this lies somewhere between these extremes, but wouldn’t know.
We also don’t know how many of these accidents were prevented by attentive drivers.
The fact that AI can give no easily understood excuses, and improvements to the algorithms is a trade secret means that any fear cannot be abated, aside from outright banning of AI driving.
Hopefully the NHTSA is also going to investigate the many other cars with traffic aware cruise control, which have the same problem with stationary vehicles.
I was driving my Rav4 with TACC enabled, and it would have crashed into a column of stopped cars if I hadn't noticed in time and intervened.
What's your denominator? Is this per year or per kilometre driven with TACC active?
Autopilot encourages people to not pay as much attention while driving, until a critical situation comes up at which point they're expected to be fully in control. That's a design flaw that isn't fixed with a liability shifting "Agree" button.
If you do this three times in a drive you have to power off and on the car again to be able to use autopilot again.
I've done maybe 40.000 km on autopilot and it's very aggressive compared to my other car (ioniq electric) when it comes to attention.
The fact that some people purposely defeat this system with weights on the wheel is another matter.
But saying that tesla encourages you to not paying attention is not correct.
All the articles with industry analysts insisting that other driver monitoring systems are nearly perfect have sometimes blinded me to that fact.
A lot of people don't like any of these systems for a lot of the same reasons that people didn't like basic cruise control in the early days.
I don't disagree with the idea of having a gradient to measure our progress up to full autonomy, but I haven't seen much to say that we should deploy the intermediate stages. Until autonomy can be reasonably guaranteed at at least human levels of safety under the assumption of no human control, it really shouldn't be deployed. "But the human had 673 milliseconds to take control and avert the accident! It's their fault!" is simply not a reasonable defense.
(There's an argument that if you take away the routine parts of driving, humans won't be able to deal with anomalous situations, but that's a different argument than the attentiveness one.)
(And, to be clear, I'm not asking for perfect safety, I'm asking for "human or better", and also understand that there may be some interesting fuzziness near the border where the two have different strengths and weakenesses but come out statistically the same. However, I don't feel like we're there yet.)
Car AP is way harder than plane AP and saying "But planes can do it, why is Tesla so bad at this" is facetious.
Yeah, and that's how it's going to be with car autopilots. Which is why it pisses me off that people want to stop progress on it.
Airplane autopilots, depending on generation, are full touch. The pilot programs the waypoints. The pilot is alerted and agrees to every course correction and change.
12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest.