News for hackers. Stuff that matters.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by txcwpalpha on Thursday June 07, 2018 @01:46PM (#17258263)
And this is exactly why all of these articles recently about how "great" it is that Tesla sends out frequent OTA updates are ridiculous. Frequent, unpredictable updates with changelogs that just read "Improvements and bug fixes" is fine when we're talking about a social media app, but is entirely unacceptable when we're talking about the software that controls a 2 ton hunk of metal flying at 70mph with humans inside of it.

The saying has been beat to death, but it bears repeating: Tesla is a prime case where the SV mindset of "move fast and break things" has resulted in "move fast and kill people". There's a reason that other vehicle manufacturers don't send out vehicle software updates willy-nilly, and it's not because they're technologically inferior.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by slg on Thursday June 07, 2018 @01:57PM (#17258405)
This isn't an issue specific to Tesla as all automakers are now making cars that are more and more dependent on software. So what is the right way to handle these updates? You mentioned a clear flaw with OTA updates, but there are also numerous advantages. For example, the recent Tesla brake software issue was fixed with an OTA update. That immediately made cars safer. Toyota had a similar problem a few years ago and did a voluntary recall. That means many of those cars with buggy brake systems were on the road for years after a potential fix was available and were driven for billions of potentially unsafe miles.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by txcwpalpha on Thursday June 07, 2018 @02:03PM (#17258461)
>This isn't an issue specific to Tesla as all automakers are now making cars that are more and more dependent on software.

Cars have been dependent on software for a long time (literally decades). This isn't something new. Even combustion engine cars have had software inside of them that controls the operation of the engine, and this software is vigorously tested for safety issues (because most car manufacturers understand a fault with such software could result in someone's death). Tesla seems to be the only major car manufacturer that has a problem with this.

>So what is the right way to handle these updates?

The way that other vehicle manufacturers (car, airplane, etc) have been doing it for decades is a pretty good way.

>You mentioned a clear flaw with OTA updates, but there are also numerous advantages. For example, the recent Tesla brake software issue was fixed with an OTA update. That immediately made cars safer.

There is no evidence that said OTA update made Tesla cars any safer. There is evidence that similar OTA updates have made Tesla cars more unsafe.

The brake OTA that you mentioned has actually potentially done more harm than good. Tesla owners have been reporting that the same update made unexpected changes to the way their cars handle/accelerate in addition to the change in braking distance. These were forced, unpredictable changes that were introduced without warning. When you're driving a 2 ton vehicle at 70mph, being able to know exactly how your car will react in all situations, including how fast it accelerates, how well it handles, how fast it brakes, and how the autopilot will act is crucial to maintaining safety. Tesla messing with those parameters without warning is a detriment to safety, not an advantage.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by omgwtfbyobbq on Thursday June 07, 2018 @04:01PM (#17259665)
>Cars have been dependent on software for a long time (literally decades). This isn't something new. Even combustion engine cars have had software inside of them that controls the operation of the engine, and this software is vigorously tested for safety issues (because most car manufacturers understand a fault with such software could result in someone's death). Tesla seems to be the only major car manufacturer that has a problem with this.

The TACC offered by most (if not all) manufacturers can't differentiate between the surroundings and stopped vehicles. I wouldn't be surprised if their Lane Keeping Assist (LKA) systems have similar problems.

https://support.volvocars.com/en-CA/cars/Pages/owners-manual...

>WARNING When Pilot Assist follows another vehicle at speeds overapprox. 30 km/h (20 mph) and changes target vehicle – from a moving vehicle to a stationary one – Pilot Assist will ignore the stationary vehicle and instead accelerate to the stored speed.

>The driver must then intervene and apply the brakes.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by darkerside on Thursday June 07, 2018 @08:24PM (#17261563)
This comparison just sold me on how morally wrong it is what Tesla is doing. Intentionally misleading and marketing to customers a feature called Autopilot that is only a marginal improvement on what other cars already offer. What if Volvo started calling their (clearly not independent) feature Autopilot and saying it was the future of hands-free driving? Seems inexcusable.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by slg on Thursday June 07, 2018 @10:59PM (#17262246)
Which is also exactly what GM is doing with Super Cruise. Here is just one of their commercials.

https://www.youtube.com/watch?v=u__51kTl4j8

Despite this warning in the manual[1]:

>Super Cruise is not a crash avoidance system and will not steer or brake to avoid a crash. Super Cruise does not steer to prevent a crash with stopped or slow-moving vehicles. You must supervise the driving task and may need to steer and brake to prevent a crash, especially in stop-and-go traffic or when a vehicle suddenly enters your lane. Always pay attention when using Super Cruise. Failure to do so could result in a crash involving serious injury or death.

[1] - https://www.cadillac.com/content/dam/cadillac/na/us/english/...

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by TeMPOraL on Friday June 08, 2018 @01:29AM (#17262834)
Riffing off the parallel thread about Google AI and how "corporations are controlled by humans" and can have moral values - no, corporations are controlled primarily by the market forces. When Tesla started branding line assist as autopilot, it put market pressure on others to follow suit. Hence, I'm absolutely not surprised about this ad and the associated warning in the manual.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by darkerside on Friday June 08, 2018 @06:14AM (#17264041)
TBF, corporations are controlled by humans and overwhelmingly influenced by market forces, which are also controlled by (other) humans.

That's a nitpick. Your broader point about Tesla pressuring the market down an unfortunate path is spot on.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by omgwtfbyobbq on Friday June 08, 2018 @05:27PM (#17269851)
That branding started a long time before Tesla was ever around.

http://www.oldcarbrochures.com/static/NA/Chrysler_and_Imperi...

Ideally, yeah, every manufacturer would have to take all the puffery out of their marketing, or better yet, talk about all the negatives of their product/service first, but I doubt I'll ever see that.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by darkerside on Friday June 08, 2018 @08:42PM (#17270858)
https://arstechnica.com/cars/2018/06/why-emergency-braking-s...

This article portrayed Super Cruise as something qualitatively different, based on the maps of existing roadways. I'm not sure if they've also considered integrating the multiple systems involved in driver assistance. I'm curious if Tesla has either for that matter.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by thinkmassive on Friday June 08, 2018 @10:31AM (#17265752)
I’m opposed to over-regulation of any sort, however it seems obvious that vehicle manufacturers need to do a better job informing consumers of the driver assistance capabilities of modern vehicles. Something similar to the health warnings on cigarette packs.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by codedokode on Thursday June 07, 2018 @05:45PM (#17260574)
> The TACC offered by most (if not all) manufacturers can't differentiate between the surroundings and stopped vehicles.

Software should not be driving a car into any of them. I think that LIDAR would see the obstacle, but as I understand, the crashed Tesla car didn't have it.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by omgwtfbyobbq on Friday June 08, 2018 @05:44PM (#17269973)
LIDAR probably would have seen the obstacle and avoided it, but so would a human driver who was operating the vehicle responsibly and correctly. It sucks that people treat level 2 systems and level 3 or 4, but the same thing applies to many convenience features in a car (cruise control, power brakes, etc...). There's always going to be some bozo doing what they shouldn't be doing with something.

I'd love to see LIDAR on consumer vehicles, but AFAIK it's prohibitively expensive. And to be fair, even Level 4 autonomous vehicles still crash into things and kill people.

https://www.bloomberg.com/news/articles/2018-05-24/uber-self...

Last but not least, every semi-autonomous system all the way back to Chrysler's "AUTO-PILOT" has had similar criticisms. People in the past even said similar things about high speed highways compared to other roads WRT attention.

http://www.curbsideclassic.com/blog/history/automotive-histo...

https://books.google.com/books?id=YpGkDAAAQBAJ&pg=PA100&lpg=...

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by friendzis on Friday June 08, 2018 @03:11AM (#17263267)
> The TACC offered by most (if not all) manufacturers can't differentiate between the surroundings and stopped vehicles.

Literally every car I have driven equipped with Cruise Control and Collision Avoidance (TACC) hits the brakes and slows down to 20-ish km/h if it senses ANYTHING moving slower (including stationary) in front of the car at possible collision path.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by ploxiln on Thursday June 07, 2018 @04:09PM (#17259744)
All cars from all manufacturers are running 10s or 100s of times more code than they did 10 or 20 years ago.

https://skeptics.stackexchange.com/questions/39559/does-the-...

This really affects the nature of the situation. 20 years ago, cars contained microcontrollers with a tiny bit of code which was thoroughly reviewed and tested by skilled professionals. Today, all cars run so much code, even outside of the entertainment system, that the review and testing just can't be the same. (And there's way more programmers, so the range of skill and care is also much wider.)

We're in a new mountain-of-flaky-software world.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by ams6110 on Thursday June 07, 2018 @04:47PM (#17260104)
When the Toyota electronic throttle "unintended acceleration" accidents were in the news, the software was described as a "big bowl of spaghetti" but the NTSB ultimately determined that it was not the cause of the problems. It was drivers using the wrong pedal.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by golfer on Thursday June 07, 2018 @10:43PM (#17262172)
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by freerobby on Friday June 08, 2018 @07:27AM (#17264298)
There are news stories like this all over because it was an ongoing saga. But the DOT's investigation showed that folks were hitting the wrong pedal:

https://www.caranddriver.com/features/its-all-your-fault-the...

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by dannyfraser on Friday June 08, 2018 @04:18AM (#17263549)
Malcolm Gladwell's podcast had a good summary of the issue, including the pedal confusion.

http://revisionisthistory.com/episodes/08-blame-game

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by dekhn on Friday June 08, 2018 @09:17AM (#17264958)
I've long been curious about the "big bowl of spaghetti" comment (and all the other criticisms made by the experts who inspected Toyota's code). There were some extremely serious accusations which don't seem consistent with the fact that the vast majority of Toyotas on the road aren't showing problems caused by their MCU's spaghetti code.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by mark-r on Thursday June 07, 2018 @10:49PM (#17262195)
AI takes it to a whole new level. Neural networks are all black box, they can't be reviewed. You feed in your training data, you test it against your test data, and just have faith that it will respond appropriately to a pattern that's slightly different than anything it's seen before. Sometimes the results are surprising.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by carlmr on Friday June 08, 2018 @12:38AM (#17262626)
That's my biggest problem with AI and neural networks. You can't really measure progress here. If you wanted the same safety standards as for every other automotive software you'd have to test drive for hundreds of thousands of kilometres after every change of parameters, because there's no way to know what has changed about the AI's behavior except for testing it thoroughly.

Compare this to classic engineering where you know the changes you've made, so you can rerun your unit tests, rerun your integration tests, check your change in the vehicle and be reasonably sure that what you changed is actually what you wanted.

The other approach to autonomous driving is to slowly and progressively engineer more and more autonomous systems where you can be reasonably sure to not have regressions. Or at least to contain your neural networks to very very specific tasks (object recognition, which they're good at), where you can always add more to your test data to be reasonably sure you don't have a regression.

I don't think we'll see too many cars being controlled by neural networks entirely, unless there's some huge advancement here. Most of the reason we see more neural networks now is that our computing power has reached the ability to train sufficiently complex NNs for useful tasks. Not because the math behind it advanced that much since the 60s.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by sokoloff on Friday June 08, 2018 @09:26AM (#17265034)
> There is no evidence that said OTA update made Tesla cars any safer.

That particular OTA update significantly shortened braking distances. [The update] cut the vehicle’s 60 mph stopping distance a whole 19 feet, to 133, about average for a luxury compact sedan. That's a safer condition, IMO, and I'm uncertain how to argue that it doesn't make the car safer.

[0] - https://www.wired.com/story/tesla-model3-braking-software-up...

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by LoSboccacc on Friday June 08, 2018 @03:00AM (#17263227)
I agree with everything you said except this:

> being able to know exactly how your car will react in all situations

If one depends on intimate knowledge of his own car for safety then he’s likely already driving outside the safety envelope of the code, which was written to provide enough safety margin for people driving bad cars from 40yr ago.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by slg on Thursday June 07, 2018 @02:12PM (#17258544)
I didn't say no car has ever relied on software. I said cars are becoming more reliant on software. I don't think that is a controversial statement. I also don't think it is controversial to say that other automakers also occasionally ship buggy code. The Toyota brake issue I mentioned in the previous post is one example.

Additionally, the argument that we should continue to handle updates this way simply because we have done it this way for decades is the laziest possible reasoning. It is frankly surprising to see that argument on HN of all places.

As for the evidence that OTA updates can make things safer, this is from Consumer Reports:

>Consumer Reports now recommends the Tesla Model 3, after our testers found that a recent over-the-air (OTA) update improved the car’s braking distance by almost 20 feet. [1]

That update going out immediately OTA is going to save lives compared to if Tesla waited for the cars to be serviced like other manufacturers. I don't think you can legitimately argue against that fact.

[1] - https://www.consumerreports.org/car-safety/tesla-model-3-get...

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by txcwpalpha on Thursday June 07, 2018 @02:15PM (#17258580)
> That update going out immediately OTA is going to save lives compared to if Tesla waited for the cars to be serviced like other manufacturers. I don't think you can legitimately argue against that fact.

There is again no evidence to support this fact. There is evidence that Tesla's OTA software updates have introduced safety issues with Tesla cars. That's a fact.

Better braking distance is of course a good thing but if anything, the fact that Teslas were on the road for so long with a sub-par braking distance is more evidence of a problem with Tesla than it is evidence of a benefit of OTA updates.

The other factor in that brake story is that it took mere days for Tesla to release an update to "fix" the brakes. This isn't a good thing. The fact that it was accomplished so quickly means that the OTA update was likely not tested very well. It also means that the issue was easy to fix, which calls into question why it wasn't fixed before. It also highlights the fact that Tesla, for some reason, failed to do the most basic testing on their own cars for braking distance. Comparing the braking distance of their cars should have been one of the very first things they did before even selling the cars, but apparently it took a third party to do that before Tesla was even aware of the issue. This doesn't inspire confidence in Tesla cars at all.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by slg on Thursday June 07, 2018 @02:18PM (#17258615)
I simply don't know what to say to you if you are going to legitimately argue that shaving 20 feet off of braking distance will not make a car any safer.

EDIT: The comment I was replying to was heavily edited after I responded. It originally said something along the lines of improving braking distance is good but there is no evidence that it would improve safety.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by Lazare on Thursday June 07, 2018 @07:33PM (#17261310)
I think you're misunderstanding the question.

> if you are going to legitimately argue that shaving 20 feet off of braking distance will not make a car any safer.

Nobody is arguing that. We're arguing that there is no evidence the Tesla OTA update made the cars safer on net.

You're trying to set up some sort of "OTA updates are dangerous in general, but this one is clearly good, how do we balance it" conversation, but the problem is, this OTA update is not clearly good. OTA updates are dangerous in general, and also in this case in specific. You need to find a better example where there's actual difficult tradeoffs being made, and not just a manufacturer mishandling things.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by rad_gruchalski on Friday June 08, 2018 @05:40AM (#17263902)
> I simply don't know what to say to you if you are going to legitimately argue that shaving 20 feet off of braking distance will not make a car any safer.

If the car can’t see the obstacle, the braking distance simply does not matter.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by txcwpalpha on Thursday June 07, 2018 @02:24PM (#17258664)
And yet again, the same OTA update changed other parameters about the way the car drives that do make it less safe. I don't know why you're trying to ignore that fact. If I drastically improve the braking distance of a car, but in the same update I also make it so that the car crashes itself into a wall and kills you, is the car safer? Hint: no

As for your edit, you clearly misread the original comment, which is why I edited it for you. I said that there was no evidence that the OTA made the car safer. Please try to read with better comprehension instead of trying to misrepresent my comments.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by hfdgiutdryg on Friday June 08, 2018 @05:59AM (#17263992)
If I drastically improve the braking distance of a car, but in the same update I also make it so that the car crashes itself into a wall and kills you, is the car safer? Hint: no

You don't have enough information to come to that conclusion.

It's quite common to have to brake hard to avoid a cousin. It's pretty uncommon to see the specific scenario triggering this crash behavior.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by slg on Thursday June 07, 2018 @02:35PM (#17258778)
I never denied that. Your comment pointed out a problem with OTA updates and I agreed calling it "a clear flaw". I pointed out a benefit of OTA updates then asked an open ended question about how they should be handled. You responded be attacking the example I provided. I was looking to debate this serious issue, not getting into a pissing match about it.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by txcwpalpha on Thursday June 07, 2018 @02:37PM (#17258809)
I never said you denied it, I said you ignored it. If you wanted to debate this serious issue, then maybe you shouldn't keep ignoring one of the crucial cornerstones of the discussion. If you're unwilling to discuss points that challenge your own opinion, then it's clear that you're just trying to push an agenda rather than have an actual discussion.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by TeMPOraL on Friday June 08, 2018 @01:26AM (#17262812)
> So what is the right way to handle these updates?

Avoid doing them in the first place? It's not like bit rot is - or should be - a problem for cars. It's a problem specific to the Internet-connected software ecosystem, which a car shouldn't be a part of.

So basically: develop software, test the shit out of it, then release. If you happen to find some critical problem later on that is fixable with software, by all means fix it, again test the shit out of it, and only then update.

If OTA updates on cars are frequent, it means someone preferred to get to market quickly instead of building the product right. Which, again, is fine for bullshit social apps, but not fine for life-critical systems.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by JimmyAustin on Friday June 08, 2018 @03:38AM (#17263397)
Tesla does test the shit out of it before they release a patch. The problem is that users expectations of the systems performance suddenly get out of sync with what the car is going to do.

Part of me wonders if there should be a very quick, unskipable, animated, easy to understand explanation of the patch notes before you can drive when they make material changes to core driving functionality.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by letsgetphysITal on Friday June 08, 2018 @04:22AM (#17263569)
While using Autopilot (Big A), there should be a loud klaxon every 30 seconds followed by a notification "CHECK ROAD CONDITIONS" and "REMAIN ENGAGED WITH DRIVING" in the same urgent tone of an aircraft autopilot (small a) warning system.

Tesla did make a mistake calling it Autopilot, but only because regular folk don't understand that aircraft autopilot is literally a heading, altitude, and speed, and will not make any correction for fault. Aircraft autopilot will fly you straight into a mountain if one happens to be in the way.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by rayiner on Friday June 08, 2018 @08:18AM (#17264557)
I don't know why Tesla defenders keep repeating this FUD:

> Tesla did make a mistake calling it Autopilot, but only because regular folk don't understand that aircraft autopilot is literally a heading, altitude, and speed, and will not make any correction for fault. Aircraft autopilot will fly you straight into a mountain if one happens to be in the way.

Auto-TCAS and Auto-GCAS exist, and the public is aware of them: E.g. http://www.airbus.com/newsroom/press-releases/en/2009/08/eas.... http://aviationweek.com/air-combat-safety/auto-gcas-saves-un....

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by rayiner on Friday June 08, 2018 @08:14AM (#17264527)
Or make certain critical updates only part of a physical recall, which provides notice to users that behavior will chance.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by mnl on Friday June 08, 2018 @05:03AM (#17263740)
Well, then what are they testing exactly?

This is beyond broken, it's a fundamental misunderstanding of how physical products are supposed to work. Software people have gotten used to dismiss the principle of least astonishment because they know better —and no user got killed because of a Gmail redesign—, but this is a car, it's hardware with its user on-board, a lot of kinetic energy and all of it relies on muscle memory.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by salawat on Friday June 08, 2018 @06:26PM (#17270241)
Obviously they don't, or the braking update would never have needed to happen in the first place.

Highways are not, nor should they ever be if at all possible, proving grounds.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by TeMPOraL on Friday June 08, 2018 @04:19AM (#17263550)
I'd vote in favor of such explanation, though this alone may not be enough to cancel out possibly thousands of hours of experience with the previous system behavior.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by mannykannot on Thursday June 07, 2018 @05:02PM (#17260239)
The first thing about doing it right is to make sure it has been developed in an appropriate manner for safety-critical systems, which includes, but is by no means limited to, adequate testing.

The second thing is to require the owners to take some action as part of the installation procedure, so that it is hard for them to overlook the fact that it has happened.

The third thing is that changes with safety implications should not be bundled with 'convenience/usability' upgrades (including those that are more of a convenience for the manufacturer than for the user.) To be fair, I am not aware of Tesla doing that, but it is a common enough practice in the software business to justify being mentioned.

And it has to be done securely. Again, I am not aware of Tesla getting this wrong.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by lttlrck on Thursday June 07, 2018 @08:58PM (#17261729)
Great that they fixed the brakes OTA. But how exactly did the inferior braking algorithm get on the Model 3 in the first place? And what are the chances of a regression?
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by Bluestrike2 on Friday June 08, 2018 @12:45AM (#17262656)
While I like Tesla, I find the praise for Tesla's fast OTA update for its braking problem to be freaking terrifying.

A problem with variable stopping distances is the sort of thing that should be blindingly obvious in the telemetry data from your testing procedures. Brake systems, and ABS controls in particular, are normally rigorously tested over the course of 12-18 months in different environments and conditions.[0] That Tesla completely missed something like that suggests either their testing procedures are drastically flawed (missing something that CR was able to easily and quickly verify in different cars), that their software development process isn't meshed up with their hardware testing and validation, or a combination of the two. Neither option is a good one.

The fact that Tesla was able to shave 19 feet of their braking distances is horrifying. After months of testing different variations and changes to refine your braking systems, shaving off an extra 19 feet should be impossible. There shouldn't be any room to gain extra inches without making tradeoffs in performance in other conditions that you've already ruled out making. If there's an extra 19 feet to be found for free after a few days of dev time, you did something drastically wrong. And that's completely ignoring physical testing before pushing your new update. Code tests aren't sufficient; you're changing physical real-world behavior, and there's always a tradeoff when you're dealing with braking and traction.

Tesla is being praised by consumers and the media because, hey, who doesn't like the idea that problems can be fixed a couple days after being identified? That's great. In this case, Tesla literally made people's cars better than they were just a few days before. But it trivializes a problem with very real consequences, and I hope that trivialization doesn't extend to Tesla's engineers. Instead of talking about a brake problem, people are talking about how great the fast OTA update for the problem is. Consumers find that comforting, as the OTA updates can makes what's otherwisea pain in the ass (recalls and dealer visits for software updates) effortless.

Hell, I'm a believer in release early, release often for software. Users benefit, as do developers. At the same time, the knowledge that you can quickly fix a bug and push out an update can be a bit insidious. It's a bit of a double-edged sword in that it gives you a sense of comfort that can bite you in the ass as it trivializes the consequences of a bug. And when bug reports for your product can literally come in the form of coroner's reports, that comfort isn't a good thing for developers.

0. https://www.forbes.com/sites/samabuelsamid/2018/05/30/rapid-...

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by pishpash on Thursday June 07, 2018 @10:43PM (#17262167)
Give me a dumb car. The software is not ready. Get them off the road.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by hfdgiutdryg on Friday June 08, 2018 @06:01AM (#17263999)
Keep them on the road. They're still safer than a huge percentage of the awful drivers that I see every day.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by salawat on Friday June 08, 2018 @06:39PM (#17270302)
They really aren't. At least you can rely on 99% of humans to try to act according to self-preservation instinct MOST of the time.

I see a Tesla, and I try to get away from them as soon as possible.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by hfdgiutdryg on Saturday June 09, 2018 @02:17PM (#17274238)
At least you can rely on 99% of humans to try to act according to self-preservation instinct MOST of the time.

Nope. I see tremendous numbers of distracted drivers who don't even realize there's a threat. I also see many utterly incompetent drivers who will not take any evasive action, including braking, because they simply don't understand basic vehicle dynamics or that one needs to react to unexpected circumstances.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by syrrim on Friday June 08, 2018 @06:27PM (#17270244)
Updates should fix problems not create new ones. The tried and true method for silicon valley bug fixing is to ship it to the users and let them report any issues. This is wholly insufficient for car software. Car software should seldom have bugs in the first place, but OTA updates should never bar never introduce new bugs to replace the old.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by amelius on Thursday June 07, 2018 @06:32PM (#17260926)
> So what is the right way to handle these updates?

Require updates to be sent to a government entity, which will test the code for X miles of real traffic, and then releases the updates to the cars. Of course, costs of this are to be paid by the company.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by daveguy on Friday June 08, 2018 @01:34AM (#17262853)
Current development of cars is done with safety as a paramount concern. There is no need to filter everything through a government entity. However the automobile companies are responsible for their design decisions. This should absolutely apply to software updates. That does mean complete transparency during investigations, a complete audit trail of every software function invoked prior to a crash.

So, no filter, but government penalties and legal remedies should be available.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by CaptainZapp on Friday June 08, 2018 @02:11AM (#17263012)
"Current development of cars is done with safety as a paramount concern."

That's exactly the impression that I don't get from Tesla very much. Instead I see the follwing:

Get that thing to market as quickly as possible. If the software for safety critical systems is sub-par, well, can be fixed with OTA updates. That's fine for your dry cleaning app. For safety critical software that's borderline criminal

Hype features far beyond their ability (autopilot). Combine this with OTAs, which potentially change the handling of something that is not at all autopilot, but actually some glorified adaptive cruise control. For good measure: Throw your customers under the bus if ineviteble and potentially deadly problems do pop up

Treating safety issues merely as a PR problem and acting acordingly. Getting all huffy and insulted and accusing the press of fake news when such shit is pointed out

I could go on. But such behavior to me is not a company signaling that safety is of paramount concern.

"That does mean complete transparency during investigations, a complete audit trail of every software function invoked prior to a crash."

Let's just say that Tesla's very selective handling and publication of crash data does not signal any inclination for transparency.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by TeMPOraL on Friday June 08, 2018 @01:38AM (#17262871)
I agree. I think companies should be losing serious money and individual should be losing jobs over crashes like these, much like in the aircraft sector.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by qaq on Thursday June 07, 2018 @07:14PM (#17261216)
Seriously ? so instead of taking x amount of time it will take x amount of time + a few years.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by amelius on Thursday June 07, 2018 @08:09PM (#17261492)
Testing is absolutely necessary. We're talking about millions of cars here, which are potentially millions of deadly weapons. You don't want companies pushing quick fixes, which turn out to contain fatal bugs.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by CamperBob2 on Thursday June 07, 2018 @07:16PM (#17261223)
That sounds like a great way to stall all further progress, which has a horrific human cost of its own.

Government has a valid role to play, though, by requiring full disclosure of the contents of updates and "improvements," by setting and enforcing minimum requirements for various levels of vehicle autonomy, and by mandating and enforcing uniform highway marking standards. Local DOTs are a big part of the problem.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by ekianjo on Thursday June 07, 2018 @06:55PM (#17261100)
Yeah, because we know governments are really good at giving certifications and doing tests that mean sonething. Lets put every design decision in the hand of governements then! or better, nationalize car companies! Problem solved?
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by AmVess on Thursday June 07, 2018 @09:50PM (#17261934)
Flying in an airplane is safe because of direct intervention by the government.

Cars have been made safe for us also by direct intervention by the government. From important things like mandating seat belts and crash safety to smaller things like forcing the recall of tens of millions of faulty air bag inflators.

These are just a few of the many things Uncle Sam has done to make things safer for us.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by erikpukinskis on Thursday June 07, 2018 @10:50PM (#17262198)
Isn’t flying mostly safe because of post hoc safety analysis followed by operating requirements? I don’t think the FAA tests every change made to aircraft before they can fly?
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by AceyMan on Thursday June 07, 2018 @11:14PM (#17262310)
De facto the FAA indeed does that very thing.

First, any change in design (or in configuration, in the case of repairs) is backed by PEs or A&P mechanics who sign off on the changes. Their career rides on the validity of their analysis so that's a better guarantee than some commit message by a systems programmer.

Second, the FAA basically says "show us what you are changing" after which they will absolutely require physical tests (static or dynamic tests, test flights, etc., as appropriate to the scope of change).

And I'd say flying is so safe mainly from the blameless post-mortem policy that the American industry instantiated decades ago and which is constantly reinforced by the pros in the NTSB. It's a wonderful model for improvement.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by sokoloff on Friday June 08, 2018 @09:37AM (#17265167)
I think that the FAA's role is theoretically as you express, but in practice, there is significantly less oversight (especially direct oversight) than implied.

As an example, the crash of N121JM on a rejected takeoff was due (only in part) to a defective throttle quadrant/gust lock design that went undetected during design and certification, in part because it was argued to be a continuation of a conformant and previously certificated design. (Which is relevant to the current discussion in that if you decide to make certification costly and time-consuming, there will be business and engineering pressure to continue using previously certificated parts, with only "insignificant changes".)

https://aviation-safety.net/database/record.php?id=20140531-...

PS: I 100% agree on the NTSB process' contribution to safety.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by erikpukinskis on Saturday June 09, 2018 @01:44AM (#17271721)
That's fascinating, it's kind of a collectivizing of the responsibility for safety.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by zaarn on Friday June 08, 2018 @04:03AM (#17263498)
If I, as an engineer, sign of on changing the screws on the flaps for cheaper ones and the plane crashes because the flaps go loose due to the screws being unable to handle the stress, my career can be assumed over if I have no good explanation.

If an engineer signs off a change they sign that they have validated all the constraints and that for all they know the machine will work within the specs with no faults.

If a software engineer commits code we may run some tests over it, look a bit over it. That's fine. But if the software ends up killing anyone, the software engineer is not responsible.

And yes, to my knowledge, every change to an aircraft is tested before flight or atleast validated by an engineer that understands what was just changed.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by amelius on Thursday June 07, 2018 @07:00PM (#17261129)
In any case, let a third party control the actual updating, so that we know when and how often cars are updated. Require at least X months of time between code submission and deployment to cars. We don't want a culture of "quick fixes".
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by stickfigure on Thursday June 07, 2018 @07:27PM (#17261279)
This is a popular idea: Just put someone in charge! It ignores the incentives for those gatekeepers, who are now part of the system. In practice I don't think you're going to get better updates, you're going to get "almost no updates".

See also: The FDA and drug approvals.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by amelius on Thursday June 07, 2018 @07:51PM (#17261394)
So you want to do away with the FDA? Put BigPharma in charge and let them regulate themselves? That sounds a lot less appealing to me.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by ekianjo on Friday June 08, 2018 @05:45AM (#17263935)
It took years for the FDA to investigate Theranos in case you are not aware. And they only did when the press started digging up. Poor, poor track record.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by wooter on Friday June 08, 2018 @12:30AM (#17262607)
fdareview.org - the fda kills more people in late/non approvals than it saves through denials. this is why compassionate use has been on the rise
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by TeMPOraL on Friday June 08, 2018 @01:36AM (#17262859)
It does not, in any way, imply that pharmaceutical companies would do a better job if left to their own devices and market pressure.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by dnautics on Friday June 08, 2018 @03:16AM (#17263290)
There's a lot of sunlight between letting pharma companies run rampant and having the FDA. One could imagine private non-profit testing and qualifications standards organizations along the lines of the underwriters laboratories
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by amelius on Friday June 08, 2018 @07:48AM (#17264402)
And who will manage all that testing? A government entity? Because that would bring us back to the original point.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by dnautics on Monday June 11, 2018 @11:07AM (#17284861)
UL is not "managed" by a government.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by ekianjo on Friday June 08, 2018 @05:47AM (#17263955)
It is not completely out of this world to imagine multiple private entities involved in pharma dossier reviews instead of having the FDA. The FDA employs tons of private consultants anyway so they bring virtually no value.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by WiseWeasel on Thursday June 07, 2018 @02:11PM (#17258533)
Certainly communication of any changes to all drivers inexperienced with the latest version; ideally user interaction required for the update to be applied, and potentially even the ability to reverse them if they are unhappy with the changes.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by LoSboccacc on Friday June 08, 2018 @02:52AM (#17263195)
At the very _least_ when you introduce a change in behavior have it to be enabled from the user through the dashboard. This creates at least one touch point for user education.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by Pfhreak on Thursday June 07, 2018 @09:29PM (#17261857)
This seems testable. IANAAE (I Am Not An Automotive Engineer), but why can't you run both the new and old code side by side and if the actions they take are materially different investigate further? Like, if in one case the new code would want you to move left, and the old code goes straight, one of those behaviors is probably wrong. If the driver corrects, then the new code is probably correct, but if the driver does not, then the new code is doing something probably incorrect.

At the very least, you should be able to get some sort of magnitude/fuzzy understanding of how frequently the new code is disagreeing, and you can figure out where and go check out those conditions.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by salawat on Thursday June 07, 2018 @10:58PM (#17262238)
It has already been touched on by another commenter, but testability of Machine Learned systems outside of a training dataset is pretty much a crapshoot.

An ML solution stops "learning" after training and only reacts.

To illustrate the difference, have you driven on roads under construction lately? As humans, when you've driven the same road hundreds of times, you start to do the same thing as a machine learned implementation. You drive by rote.

When you get to that construction zone though, or the lines get messed up, your brain will generally realize something has changed, and you'll end up doing something "unpredictable", i.e. learning a new behavior. The Machine Learning Ali's output (a neural net) can't do that. It can generify a pattern to a point, but it's behavior in truly novel circumstances cannot be assured.

Besides which, the problem still stands that the system is coded to ignore straight ahead stationary objects. Terrible implementation. It should look for overly fast and uniform increase in angular field coverage combined with being near stationary in terms of relative motion as a trigger to brake. I.e. If a recognized shape gets bigger at the same rate on all "sides" whilst maintaining a weighted center at the same coordinate. It's one of the visual tricks pilots are taught to avoid mid-air collisions.

Admittedly though, the human brain will likely remain WAY better at those types of tricks than a computer will be for a good long time.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by mortehu on Friday June 08, 2018 @12:39AM (#17262631)
I think the point is to run the two models on the same data, either simultaneously in everyone's cars, or using recorded sensor data from the past (maybe in the car while parked after a drive for privacy reasons). Initially, only the old version gets to take any action in the real world. Any difference in lane choice would then have to be justified before making the new version "live".

You can do this sort of stuff when replacing a web service too, by the way. For example running two versions of Django and checking if the new version produces any difference for a week before making it the version the client actually sees.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by salawat on Friday June 08, 2018 @01:27AM (#17262819)
The problem, however, is your test can't be assumed to generify the way an explicitly coded web service would.

You can look at your code and say "For all invalid XML, this, for all input spaces, that." You can formally prove your code in other words.

You CANNOT do that with Neural Nets. Any formal proof would simply prove that your neural network simulation is still running okay. Not that it is generating the correct results.

You can supervise the learning process, and you can practically guarantee all the cases within your training data set, and everyone in the research space is comfy enough to say "yeah, for the most part this will probably generify" but the spectre of overfitting never goes away.

With machine learning, I developed a rule of thumb for applicability: "Can a human being who devotes their life to the task learn to do it perfectly?"

If the answer is yes, it MAY be possible to create an expert system capable of performing the task reliably.

So lets apply the rule of thumb:

"Can a human being, devoting their life to the task of driving in arbitrary environmental conditions, perfectly safely drive? Can he safely coexist with other non-dedicated motorists?"

The answer to the first I think we could MAYBE pull off by constraining the scope of arbitrary conditions (I.e. specifically build dedicated self-driving only infrastructure).

The second is a big fat NOPE. In fact, studies have found that too many perfectly obedient drivers typically WORSEN traffic in terms of probability to create traffic jams. Start thinking about how people drive outside the United States and the first-world in general, and the task becomes exponentially more difficult.

The only things smarter than the engineers trying to get your car to drive itself are all the idiots who will invent hazard conditions that your car isn't trained to handle. Your brain is your number one safety device. Technology won't change that. You cannot, and should not outsource your own safety.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by YeOakAye on Friday June 08, 2018 @05:27AM (#17263847)
Edge cases in this scare the hell out of me. I'm envisioning watching CCTV of every Tesla that follows a specific route on a specific day merrily driving off the same cliff until it's noticed.

I mean what would have happened here if another Tesla or two were directly behind Huang, following his car's lead?!

Possibly nothing, I'd assume the stopping distance would be observed and the following cars would be able to stop/avoid, but I wouldn't like to bet either way. Perhaps, in some conditions, the sudden impact on the lead car would cause the second car to lose track of the rear end of the first? Would it then accelerate into it?

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by salawat on Friday June 08, 2018 @09:28AM (#17265053)
The report indicates the system ignores stationary objects. I would not be surprised if a suddenly decelerated car in front of the system effectively vanished from the car's situational awareness. Your scenario does not seem that far-fetched.

An attentive human would realize something horrible had happened and perhaps reacted accordingly. A disengaged or otherwise distracted one may not have the reaction time necessary to stop the system from plowing right into the situation to make it worse.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by dominotw on Thursday June 07, 2018 @11:38PM (#17262398)
Basically AI is a sophisticated ' hotdog not hotdog'.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by akira2501 on Thursday June 07, 2018 @10:00PM (#17261974)
> but why can't you run both the new and old code side by side and if the actions they take are materially different investigate further?

Because there are no closed facilities that you can use to actually perform any meaningful test. You could test "in-situ", but you would need an absolutely _huge_ testing area in order to accurately test and check all the different roadway configurations the vehicle is likely to encounter. You'll probably want more than one pass, some with pedestrians, some without, some in high light and some in low, etc..

It's worth noting that American's drive more than 260 billion miles each _month_. It's just an enormous problem.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by dchichkov on Thursday June 07, 2018 @10:27PM (#17262087)
This particular case might have been testable "in-situ":

[ABS News Reporter] Dan Noyes also spoke and texted with Walter Huang's brother, Will, today. He confirmed Walter was on the way to work at Apple when he died. He also makes a startling claim — that before the crash, Walter complained "seven-to-10 times the car would swivel toward that same exact barrier during auto-pilot. Walter took it into dealership addressing the issue, but they couldn't duplicate it there."

It is very believable that the car would swivel toward the same exact barrier on auto-pilot.

BTW - I'm running a nonprofit/public dataset project aimed at increasing safety of autonomous vehicles. If anyone here wants to contribute (with suggestions / pull requests / following it on twitter / etc) - you'd be most welcome. Its: https://www.safe-av.org/

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by yogrish on Thursday June 07, 2018 @11:05PM (#17262277)
This is where simulators play an important role. Many AD( automated driving) solution suppliers are investing into simulators to create different scenarios and test performance of their sensors and also SW. Else, as you said it’s impossible to drive billions of miles to cover all usecases.
Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by jakecopp on Thursday June 07, 2018 @10:00PM (#17261972)
A/B testing self-driving car software to find bugs? Would probably work great, but that is also terrifying!

But you're right, if you're really going to do a full roll out, may as well test it on a subsegment first - I'd hate for it to be used as a debugging tool though.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by taneq on Friday June 08, 2018 @01:21AM (#17262795)
> why can't you run both the new and old code side by side and if the actions they take are materially different investigate further?

As I understand it, this is essentially what they were doing with the autopilot 'shadow mode' stuff. Running the system in the background and comparing its outputs with the human driver's responses, and (presumably) logging an incident when the two diverged by any significant margin?

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by InternetOfStuff on Friday June 08, 2018 @04:57AM (#17263715)
> And this is exactly why all of these articles recently about how "great" it is that Tesla sends out frequent OTA updates are ridiculous. Frequent, unpredictable updates with changelogs that just read "Improvements and bug fixes" is fine when we're talking about a social media app, but is entirely unacceptable when we're talking about the software that controls a 2 ton hunk of metal

I recently got downvoted for that exact line of reasoning.

Looks like some people don't like to hear that :-)

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by bertil on Friday June 08, 2018 @09:53AM (#17265353)
I think you got down-voted because you misrepresented what Telsa is actually doing, which is a difficult arbitrage between:

- known preventable deaths from, say, not staying in the lane aggressively enough;

- possible surprises and subsequent deaths.

There is an ethical conundrum (that is quite different from the trolley problem) between a known fix and an unknown possibility. If both are clearly establish and you dismiss the former to have a simplistic take, yes, you would be down-voted because you are making the debate less informed.

Without falling to solutionism, in that case, the remaining issue seems rather isolated to a handful of areas that look like lanes and could be either painted properly, or that Telsa cars could train to avoid. The later fix would have to be sent rapidly and could have surprising consequences -- although it seems decreasingly likely.

That learning pattern (resolving unintended surprises as they happen decreasingly often) is common in software. This explains why this community prefers (to an extend) Telsa to other manufacturers. Others have preferred the surprise-free and PR-friendly option of not saving the dozens of thousand of lives dying on the road at the moment. There are ethical backgrounds to non-interventionism.

As the victim of car violence, I happen to think that their position is unethical. I’m actually at what is considered an extreme position, of being in favour of Telsa (and Waymo) taking more risk than necessary and temporarily increasing the number of accidents on the road because they have a far better track record of learning from those accidents (and the subsequent press coverage) and that would lower the overall number of deaths faster.

As it happens, they don’t need to: even with less than half of accidents from their counterparts, they still get spectacular learning rate.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by InternetOfStuff on Tuesday June 12, 2018 @07:37AM (#17292346)
Thanks for your comment. I think you may be right.

> I think you got down-voted because you misrepresented what Telsa is actually doing, which is a difficult arbitrage between: > > - known preventable deaths from, say, not staying in the lane aggressively enough; > > - possible surprises and subsequent deaths.

I don't think I was misrepresenting anything (at least, I was trying not to). I just pointed out that behaviour-changing updates that may be harmless in, say, smartphone apps, are much more problematic in environments such as driving-assisted cars. I think this is objectively true. And I think we need to come up with mechanisms to solve these problems.

> That learning pattern (resolving unintended surprises as they happen decreasingly often) is common in software.

My argument is that changes in behaviour are (almost automatically) surprising, and thus inherently dangerous. Unless my car is truly autonomous and doesn't need my intervention, it must be predictable. Updates run the risk oof breaking htat predictability.

> Others have preferred the surprise-free and PR-friendly option of not saving the dozens of thousand of lives dying on the road at the moment.

My worry is that (potentially) people will still die, just different ones.

> being in favour of Telsa (and Waymo) taking more risk than necessary

If I'm taking you literally, that's an obviously unwise position to take ("more than necessary"). But I think I know what you meant to say: err on the side of faster learning, accepting the potential consequences. Perhaps like NASA in the 1960s.

But my argument was simply that there is a problem with frequent, gradual updates. Not that we shouldn't update (even though that's actually one option). We ought to search for solutions to this problem. I can think of several that aren't "don't update".

But claiming that the problem doesn't exist, or that those that worry about it are unreasonable, is unhelpful.

Re:NTSB: Autopilot steered Tesla car toward traffic barrier ... by erikpukinskis on Thursday June 07, 2018 @10:35PM (#17262129)
More accurately: move fast and save some lives and kill people.