News for hackers. Stuff that matters.
Re:Uber disabled Volvo SUV's safety system before fatality by kelnos on Monday March 26, 2018 @11:23PM (#16684941)
It seems a little weird to focus on this. The article itself admits it may be standard practice for makers of self-driving tech to disable any built-in automated driving features in the car. This makes sense: presumably there's no good way to talk to the built-in system; having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable.
Re:Uber disabled Volvo SUV's safety system before fatality by Lazare on Tuesday March 27, 2018 @12:06AM (#16685146)
> It seems a little weird to focus on this.

I'm not sure I'd say that this is being "focused" on. It's an interesting bit of info, the other players in the space (Intel, Volvo, Aptiv) have an interest in getting their spin out there, readers are interested, so we get this article and everyone is happy. It's not a super critical key to the story, but it does help explain what happened in some small way.

> having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe.

That seems like a bit of a leap to me. The Volvo system is not super-advanced; the relevant bit here would probably be their "Collision Warning with Auto Brake" feature.

Multiple independent systems with the ability to sound an alarm and slam on the brakes if they detect an imminent pedestrian collision...seem like they probably would be safer overall? They'd certainly be safer for the pedestrians, and we're currently at 1 pedestrian fatality and 0 fatalities for "people in cars killed due to overzealous/conflicting automated safety systems".

I rather suspect Uber disabled it because they wanted better data, not because they thought it was safer for other people on the road. That's certainly a valid choice, but one which might have been wiser if their tech was a lot more reliable.

Re:Uber disabled Volvo SUV's safety system before fatality by jschwartzi on Tuesday March 27, 2018 @06:27AM (#16686949)
Disabling a backup system like that strikes me as very negligent from an engineering perspective. They had reason to expect their system to fail because it's a prototype, but then they go and disable another system that is actively marketed and that could have prevented this death.

I'm so glad they're not testing their cars in my state. This makes me wonder if they even do basic safety engineering like FMEA or if they just wing it.

Re:Uber disabled Volvo SUV's safety system before fatality by Already__Taken on Tuesday March 27, 2018 @07:45AM (#16687373)
But it was never built as a backup system. It's a whole feature that was designed from the ground up assuming it's the only thing on the car doing anything like the job it does.

There's 2 chefs.

Re:Uber disabled Volvo SUV's safety system before fatality by Lazare on Tuesday March 27, 2018 @09:10AM (#16687918)
> But it was never built as a backup system.

The collision avoidance system? The fail-safe that's meant to ensure the car doesn't hit something when the human driver screws up? The one that works with an entirely separate autonomous intelligence (aka, the human) controlling the car? The one explicitly designed to trigger emergency brakeing even if the primary controller is trying to do something else? The kind of system installed on many other AI controlled cars to avoid this exact type of disaster?

That backup system?

Look, it's entirely possible that for some reason this specific safety system conflicted with the current version of Uber's system, but we have zero data to suggest that, and if so, it's something specific to those two systems in that particular vehicle (and suggests a severe issue with Uber's system to boot). There's no general issue here.

> There's 2 chefs.

Yes, and sometimes one of them takes over from the other one. That's how this works.

Re:Uber disabled Volvo SUV's safety system before fatality by BoorishBears on Tuesday March 27, 2018 @09:23AM (#16688032)
Have you actually driven with one of these systems?

There are plenty of intentional maneuvers required to integrate with real-world (ie. non perfect) traffic that trigger these systems.

They’re designed to take hints from human input (so for example, LKA responds less strongly to forceful unexpected inputs than a lack of expected input).

There is 0 chance the system wouldn’t interfere with a self driving.

Not to mention many of those safety systems include lockouts on the CAN bus which make removing their safety module a first step to even add your own inputs. I know that’s how it works on my Volt with a Mobileye sensor

Re:Uber disabled Volvo SUV's safety system before fatality by tzakrajs on Tuesday March 27, 2018 @11:18AM (#16689302)
The self driving should be capable of mitigating this risk by adapting to systems like these in general.
Re:Uber disabled Volvo SUV's safety system before fatality by fixermark on Tuesday March 27, 2018 @11:31AM (#16689457)
Problem is, that greatly complicates the problem-solviing process for SD vehicles---you're now trying to build atop a model of the world provided by the vendor, with its own faults and quirks (and you probably need an agreement with the vendor to see their code / dissect their model, and now your release schedule depends on their release schedule, etc.). It also makes Uber dependent upon Volvo's buy-in on the project, and that arrangement is fraught in a world where we don't yet know if SD cars are going to damage the business of traditional auto manufacturers 50 years out---Uber wants to be in a place where if a vendor relationship sours, they can pack up and change chassis without a complete overhaul of the SD logic.

If your SD system is supposed to be solving 100% of the same problem anyway, it's best to gut the software you don't control and work with the hardware as directly as you can. The notion of building atop a lower-level abstraction makes for great academia but doesn't work with industrial realities.

Re:Uber disabled Volvo SUV's safety system before fatality by tzakrajs on Tuesday March 27, 2018 @12:00PM (#16689764)
Thanks for the explanation and I agree that it won't be as easy to build SD for auto braking systems that change from chassis to chassis, but this is what I think people want. People want to have fail safes from different vendors.
Re:Uber disabled Volvo SUV's safety system before fatality by PinguTS on Tuesday March 27, 2018 @08:21AM (#16687568)
Actually, those systems are designed as backup. They are designed as backup, when the driver does not react. That is the reason why they warn first and initiate an emergency brake only, and only if an accident becomes unavoidable.

So that would, if the automated tech from Uber would have detected in time, it would have reacted like a human driver and the pre-installed system would may have not noticed it. But in case, it would have reacted as a backup.

Re:Uber disabled Volvo SUV's safety system before fatality by kylegordon on Tuesday March 27, 2018 @10:10AM (#16688519)
On the two occasions my Volvo has decided to take over (both admittedly due to overhanging bushes in a parking space), it has taken over control immediately and entirely.

There is no warning.

Re:Uber disabled Volvo SUV's safety system before fatality by tzakrajs on Tuesday March 27, 2018 @11:19AM (#16689315)
No warning necessary because self driving cars are completely capable of being programmed to deal with varying ability to control the car. It can still have an intention whether or not the car is allowing it to proceed. It needs to make a best effort, allowing failure when a backup system takes control.
Re:Uber disabled Volvo SUV's safety system before fatality by kwhitefoot on Tuesday March 27, 2018 @11:37AM (#16689528)
Automatic emergency braking is quite definitely a backup system. If it ever trips in my car it is because the primary system, that is me, has failed to brake in time.
Re:Uber disabled Volvo SUV's safety system before fatality by majewsky on Tuesday March 27, 2018 @05:50AM (#16686796)
> we're currently at 1 pedestrian fatality and 0 fatalities for "people in cars killed due to overzealous/conflicting automated safety systems".

Not trying to take sides here, but the last part strikes me as "citation needed".

Re:Uber disabled Volvo SUV's safety system before fatality by Lazare on Tuesday March 27, 2018 @06:52AM (#16687063)
Sorry, I guess I left out the implicit "as far as I know".

Do you know of anyone who's died that way?

Re:Uber disabled Volvo SUV's safety system before fatality by dotsh on Tuesday March 27, 2018 @08:00AM (#16687445)
Re:Uber disabled Volvo SUV's safety system before fatality by majewsky on Tuesday March 27, 2018 @08:51AM (#16687777)
That's neither a "overzealous/conflicting" system nor a "safety system", so if anything, this would bring the score to 2:0 rather than 1:1.
Re:Uber disabled Volvo SUV's safety system before fatality by Lazare on Tuesday March 27, 2018 @09:14AM (#16687965)
I was already aware of that one, but sure, that's another example of the opposite of what I asked for. Rather than someone killed by conflicting or over-cautious safety systems, that's an example of someone killed by a single ineffective safety system, just like the pedestrian in Arizona.

This further reinforces my point, I think. :)

Re:Uber disabled Volvo SUV's safety system before fatality by w0m on Tuesday March 27, 2018 @09:15AM (#16687970)
> Multiple independent systems with the ability to sound an alarm and slam on the brakes if they detect an imminent pedestrian collision...seem like they probably would be safer overall?

While yes - it would also force Uber to permanently partner with Volvo on technology they can't control. 'Uhuh, my Ubers a Chevy not a Ford - it could kill me because the Chevies don't built-in X is worse!'.

You do not want an arbitrary third party system controlling the car while the Uber system is in unknown ways; if the systems conflicted things could go out of control and be more dangerous.

Re:Uber disabled Volvo SUV's safety system before fatality by mannykannot on Tuesday March 27, 2018 @11:06AM (#16689165)
> it would also force Uber to permanently partner with Volvo on technology they can't control.

That sounds like a win-win to me.

>You do not want an arbitrary third party system controlling the car while the Uber system is in unknown ways; if the systems conflicted things could go out of control and be more dangerous.

I find it interesting that, of these two systems, it is Volvo's widely-deployed system that is described as 'arbitrary'. If this argument had any merit, it would be unsafe for this system to be on the road, 'conflicting' with the car's human driver. On the contrary, the statistics clearly show that these systems reduce the frequency and severity of accidents.

Re:Uber disabled Volvo SUV's safety system before fatality by w0m on Tuesday April 03, 2018 @11:46AM (#16745784)
I know this sounds bad, but the arbitrary is that the Ubers aren't designed to be as safe as possible currently - they're designed to be testbeds of the new fully autonomous technology uber is developing.

Maybe it makes sense to rely on arbitrary vehicle specific fallbacks, but long-term it doesn't as I'm sure They don't want to be dependent on a specific model. When you request a ride, do you want to care about whether your getting a Honda or a Volvo?

Re:Uber disabled Volvo SUV's safety system before fatality by romwell on Tuesday March 27, 2018 @12:42AM (#16685303)
>having a second system running that can't coordinate with the self-driving system

... is called redundancy and is the basis of building mission-critical, safe systems.

Re:Uber disabled Volvo SUV's safety system before fatality by kelnos on Tuesday April 03, 2018 @08:00PM (#16750643)
Redundancy is when the primary system fails and the backup system takes over. Since the primary and backup in this case are unrelated, created by different entities, and the entity building the primary system may not be able to fully understand conditions that will cause the backup system to engage, this isn't so much a "backup" as a "second system that could interfere in unpredictable ways".
Re:Uber disabled Volvo SUV's safety system before fatality by TomMarius on Tuesday March 27, 2018 @03:29AM (#16686146)
No, redundancy means making a backup of your primary system so it can take over, not having a completely different uncooperative system active at the same time.
Re:Uber disabled Volvo SUV's safety system before fatality by romwell on Tuesday March 27, 2018 @03:45AM (#16686238)
This is just one of many kinds of redundancy. Please see [1] for several ways in which redundancy is manifested (e.g. triple modular redundancy).

Specifically, uncooperative systems running at the same time was exactly how fault tolerance was achieved on the space shuttle[2]:

>For true redundancy to take place, all computers must listen to all traffic on all buses, even though they might be commanding just a few. That way they know about all the data generated in the current phase. They must also be processing that data at the same time the other computers do. If there is a failure [...], since the Shuttle's other three computers are sending apparently correct commands to their actuators, the failed computer's commands are physically out-voted

That's just one example. There's no need for the definition of the word "redundancy" to be as narrow as you said.

[1]https://en.wikipedia.org/wiki/Redundancy_(engineering)

[2]https://history.nasa.gov/computers/Ch4-4.html

Re:Uber disabled Volvo SUV's safety system before fatality by TomMarius on Tuesday March 27, 2018 @03:47AM (#16686247)
Redundancy definitely does not mean running two different conflicting systems. Three (identical) - that's a different story.
Re:Uber disabled Volvo SUV's safety system before fatality by InternetOfStuff on Tuesday March 27, 2018 @05:59AM (#16686834)
But they aren't conflicting: they have the same goal (not running into stuff).

In fact, there's yet another different, redundant, potentially conflicting safety system: the human safety driver.

If only one were to trigger, it still wouldn't be a conflict.

And yeah, redundancy through using non-identical systems is very much a thing, and considered much safer than having two identical things (which might make the same mistakes).

Source: I've worked on ASIL D safety-critical automotive systems, where diverse redundant subsystems were actually a design criterion.

Re:Uber disabled Volvo SUV's safety system before fatality by simion314 on Tuesday March 27, 2018 @04:26AM (#16686419)
Why not, I have 2 different systems that detect an imminent impact in a wall, then if the systems do not agree you stop, if your main system is experimental then having the backup system that is using different tech then it seems obvious that is safer to stop, then analyze the cause of the conflict.
Re:Uber disabled Volvo SUV's safety system before fatality by slowmotiony on Tuesday March 27, 2018 @05:03AM (#16686585)
So if one of these fails, the car stops? That doesn't seem like redundancy.
Re:Uber disabled Volvo SUV's safety system before fatality by tialaramex on Tuesday March 27, 2018 @06:48AM (#16687046)
Yes, it's redundantly _safe_, which is the thing we actually care about. Safety is job #1. Well, not at Uber apparently.

There are trade-offs here, sometimes we might decide to take a very small risk, after weighing it up. A famous example is ETOPS. Unlike propeller engines, jet engines for airliners are ridiculously reliable once running. ETOPS (jokingly "Engines Turn Or Passengers Swim") is a scheme which over the past few decades has gradually allowed twin engine jets (which obviously just become over-engineered gliders if both their engines fail) to fly over oceans and other large distances with no airports, on the rationale that although total engine failure would probably be deadly without anywhere to put the plane down, two unrelated sequential engine failures are now so unlikely as to be overall negligible risk to the passengers.

But that needs a risk we understand. If you have no idea what the risk is, you can't make that trade, so redundantly safe systems are the obvious choice.

Re:Uber disabled Volvo SUV's safety system before fatality by cdancette on Tuesday March 27, 2018 @05:35AM (#16686727)
The question is not about failing but detecting an obstacle and braking.

If one system detects an obstacle, stop the car. Even if this system was failing and the other wasn't detecting anything.

And if one system fail, yes the car should stop if it's not safe to drive with only one of them.

Re:Uber disabled Volvo SUV's safety system before fatality by jschwartzi on Tuesday March 27, 2018 @06:34AM (#16686982)
In engineering we don't rely on our feeling. We have to actually have reason and evidence. You're approaching it as if the car should be moving at all costs, but the responsible engineering task is to keep the car moving unless the automated systems indicate that it's hazardous to others.
Re:Uber disabled Volvo SUV's safety system before fatality by tux1968 on Tuesday March 27, 2018 @05:32AM (#16686705)
If you walk into a store and don't see that it is being robbed at gunpoint, it would be very nice to hear the assailant ordering the cashier to empty the til. You have two chances to steer clear.

Either your eyes or ears may fail you now and again, but you're better off with both.

No?

Re:Uber disabled Volvo SUV's safety system before fatality by jakeogh on Tuesday March 27, 2018 @05:48AM (#16686789)
Nothing wrong with having a chance to steer clear, but it strikes me as an odd reaction unless you live in a society where only the criminals can defend themselves.
Re:Uber disabled Volvo SUV's safety system before fatality by m_eiman on Tuesday March 27, 2018 @05:38AM (#16686739)
Seems like a way to fail safe, which is what you should want.
Re:Uber disabled Volvo SUV's safety system before fatality by jakeogh on Tuesday March 27, 2018 @06:07AM (#16686867)
unless the obstacle appears half way through an intersection where only one direction stops?
Re:Uber disabled Volvo SUV's safety system before fatality by simion314 on Tuesday March 27, 2018 @08:38AM (#16687678)
Did you skip the part where one of the system detects hitting a wall? Why is not the normal thing to slow down and stop? You will trust the experimental system that is in testing over a safer system?
Re:Uber disabled Volvo SUV's safety system before fatality by lambda on Tuesday March 27, 2018 @10:21AM (#16688638)
It's called a failsafe. The failure mode should be the safest option; which is for the car to stop or slow until the human driver takes over.
Re:Uber disabled Volvo SUV's safety system before fatality by JorgeGT on Tuesday March 27, 2018 @05:53AM (#16686809)
The human safety driver IS a completely different uncooperative system active at the same time.
Re:Uber disabled Volvo SUV's safety system before fatality by jschwartzi on Tuesday March 27, 2018 @06:31AM (#16686965)
No, in safety-critical systems redundancy means having a backup system or systems that can support the safety-critical aspects of the system if the main control system fails.
Re:Uber disabled Volvo SUV's safety system before fatality by mannykannot on Tuesday March 27, 2018 @10:57AM (#16689036)
Only yesterday there was this discussion [1], in which the general consensus was that the safe way to do things is to put the 'smart' goal-oriented driving stuff on top of the 'dumber-but-reliable' technology to handle specific dangers. It is the latter that has produced most of the statistical evidence justifying the claim that automation makes us safer.

I am very skeptical of your claim that this is a less safe way of doing things. There is, in fact, communication between the two systems: the safety system 'says' to the driving system, "no you don't", and that should not be up for negotiation any more than it is when a human is driving. As was pointed out in the other discussion, if the driving system cannot find a way to work within the restrictions of the safety systems, it is doing it wrong.

[1] https://news.ycombinator.com/item?id=16681611

Re:Uber disabled Volvo SUV's safety system before fatality by thrownblown on Tuesday March 27, 2018 @11:20AM (#16689319)
Isn't this how the human control systems are? Without reading the referenced convo, I can see many parallels.

Like we have reflexes that we don't have full control over, that often don't even interact with our brain but rather are initiated from the spinal cord, which I would classify as dumb in this instance. And layered on top are emotional responses, memories and finally consiousness as, arguably, the control layer.

Re:Uber disabled Volvo SUV's safety system before fatality by kelnos on Tuesday April 03, 2018 @08:02PM (#16750652)
Having a dumber system underneath the smarter system is a great idea. I'm not arguing against that. I'm arguing against having two separate systems that are unrelated and are built (likely) without a deep understanding of how the other behaves.

Uber themselves should have a dumber system built underneath the smarter system (and maybe they do!); that way they understand all the variables that govern the operation of both and can be reasonably sure that the two systems won't fight each other. They can also build an interlock that causes the smarter system to completely disengage if the dumber system has to take over. I of course don't know this for a fact, but I wouldn't be surprised if Uber doesn't and can't have a complete understanding of how the Volvo system works. Disabling it makes sense. Again, they should replace it with their own dumber system, but that's a separate issue.

Re:Uber disabled Volvo SUV's safety system before fatality by dharmab on Tuesday March 27, 2018 @11:18AM (#16689301)
You could also record the decisions of the self-driving system and replay the scenario in a simulation environment (either virtual or closed course). Keep the stock safety features on for public roads and still get useful data.
Re:Uber disabled Volvo SUV's safety system before fatality by neilalexander on Tuesday March 27, 2018 @05:11AM (#16686621)
> presumably there's no good way to talk to the built-in system; having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe.

Honestly, this is a cop-out. A substantial amount of information is available from the vehicle's CAN networks - including speed and braking action!

If the autonomous driving system detects sudden and unplanned changes in speed and/or braking (perhaps caused either by the driver, an external safety system or some kind of impact with something on the road) then the obvious choice there is for the autonomous driving system to release control of the vehicle and let whatever existing systems in the car, or even the driver, handle what is happening.

The car's built-in safety systems are a) simpler, b) less likely to malfunction and c) substantially better road-tested. There was no reason whatsoever to disable them.

Re:Uber disabled Volvo SUV's safety system before fatality by davrosthedalek on Tuesday March 27, 2018 @06:35AM (#16686985)
The only reason I could imagine if the built-in system interferes with the system-under-test. For example, if the built-in LIDAR messes up the LIDAR of the AI.
Re:Uber disabled Volvo SUV's safety system before fatality by tzakrajs on Tuesday March 27, 2018 @11:27AM (#16689412)
There is no way Volvo is equipping cars with LIDAR for this kind of use case as radar will do for much much cheaper.
Re:Uber disabled Volvo SUV's safety system before fatality by rocqua on Tuesday March 27, 2018 @05:38AM (#16686737)
If the volvo system were to kick in and brake, the uber system shouldn't relinquish all control. It should only give up the brakes, if you are in the middle of a corner, you'd want the self-driving system to keep steering.
Re:Uber disabled Volvo SUV's safety system before fatality by autogeek on Tuesday March 27, 2018 @06:34AM (#16686979)
> to disable any built-in automated driving features in the car

Automatic Emergency Braking (AEB), which is the system that could have prevented this accident, is not an automated driving function though; it is an active safety function. As such, it is more common to stack such functions. For instance, ABS > AEB > automated driving, with the left most being the function with the highest priority, i.e., it can override the functions to the right. This is how most safety systems in cars work in production.

Re:Uber disabled Volvo SUV's safety system before fatality by ilamont on Tuesday March 27, 2018 @09:49AM (#16688292)
“We don’t want people to be confused or think it was a failure of the technology that we supply for Volvo, because that’s not the case,” Zach Peterson, a spokesman for Aptiv Plc, said by phone. The Volvo XC90’s standard advanced driver-assistance system “has nothing to do” with the Uber test vehicle’s autonomous driving system, he said. Aptiv is speaking up for its technology to avoid being tainted by the fatality involving Uber ...

No. This is Volvo pressuring Aptiv to make a statement on its behalf, to avoid being tainted by this PR disaster and to shift investigators' focus away from both companies.

Also, in all of the discussions about this incident, let's forget that Elaine Herzberg, a mother of two children, lost her life due to an engineering experiment and publicity stunt ("Come to Arizona! We don't have safety regs!") being conducted on public roads with great risk to members of the public. Volvo, Aptiv, and other participants in these trials should have never signed off ... yet they willingly participated, and an innocent member of the public has paid with her life.

Re:Uber disabled Volvo SUV's safety system before fatality by pythonaut_16 on Tuesday March 27, 2018 @10:08AM (#16688486)
I don't see how Volvo is responsible for Uber's recklessness. Volvo didn't make them test on public roads with unsafe technology.
Re:Uber disabled Volvo SUV's safety system before fatality by SkyPuncher on Tuesday March 27, 2018 @11:09AM (#16689207)
Critical thought would agree. Unfortunately for Volvo, the vehicle was a Volvo and without proper context (e.g. only looking at an image) it'd be easy to miss that Uber's self-driving technology was being used - not Volvo's.
Re:Uber disabled Volvo SUV's safety system before fatality by dagw on Tuesday March 27, 2018 @10:12AM (#16688535)
The question isn't if Volvo is in any way culpable (they almost certainly aren't) but if there's a risk that Volvo might appear in any way culpable to the general public. The very last thing Volvo wants is the headline "Self-driving Volvo kills pedestrian".
Re:Uber disabled Volvo SUV's safety system before fatality by chillingeffect on Tuesday March 27, 2018 @10:23AM (#16688658)
Especially because a Volvo plowed into a pedestrian almost a year ago:

https://www.youtube.com/watch?v=WbW2UgmjJUA

IIRC it was discovered this happened because the owner hadn't purchased the "Pedestrian Avoidance Package".

And shortly before that, a Volvo plowed into a parked truck (in a demo):

https://youtu.be/aNi17YLnZpg

This was "was the result of an earlier flat battery that temporarily disabled the system due to low voltage" [0]

The pattern here is not only due to the collision systems themselves, but to the system around the collision avoidance system.

[0] https://jalopnik.com/5648126/volvo-pedestrian-avoidance-cras...

Re:Uber disabled Volvo SUV's safety system before fatality by pythonaut_16 on Tuesday March 27, 2018 @12:42PM (#16690267)
To be clear, I was specifically responding to the parent who somehow acted like Volvo was somehow complicit in this crash and thus wrong to distance itself.
Re:Uber disabled Volvo SUV's safety system before fatality by pvaldes on Tuesday March 27, 2018 @11:35AM (#16689502)
> let's forget that Elaine Herzberg, a mother of two children, lost her life due to an engineering experiment

I'll sound harsh, but to be honest it seems that Elaine Herzberg's behaviour also played a role in this drama.

The video is a perfect example of jaywalking.

Dressed in black, crossing the street at night in a dark place, no lights, not reflective jacket or zebra crossing area, not paying enough attention to traffic. In the video she's not even looking at the car. There is not signs of the woman stopping or changing their trajectory to avoid the crash.

Re:Uber disabled Volvo SUV's safety system before fatality by mkempe on Tuesday March 27, 2018 @10:27AM (#16688695)
Are you saying that Volvo and Aptiv should have stopped Uber from running these tests in Arizona, because Uber disabled Volvo's proven safety features?
Re:Uber disabled Volvo SUV's safety system before fatality by la_oveja on Tuesday March 27, 2018 @09:57AM (#16688367)
Yes, it's tragic. but she crossed the road where she shouldn't. She made a mistake, and a tragic indicent happened. And like this, everyday.
Re:Uber disabled Volvo SUV's safety system before fatality by lambda on Tuesday March 27, 2018 @10:18AM (#16688590)
There were multiple failures here. Let's not put all of the blame on the victim.

This particular location has the sidewalk end on one side, and there is a paved section in the median which looks like it is intended for crossing. There is a small sign that indicates that you should not cross here.

From the video, we can see that while there were street lights on the road, the area in which the apparent crossing existed was completely dark. A pedestrian may not have been able to see the sign indicating that you shouldn't cross here; or even if she did, there are so many roads that are built in pedestrian unfriendly ways that sometimes you just have to cross at unmarked crossings to avoid having to go substantially out of your way.

Additionally, the video shows that the driver was distracted immediately before the crash. While the dash cam video seems to indicate that you can't see the pedestrian until it was probably already too late, human eyes are a lot more sensitive over a greater dynamic range than most video cameras, so there's a good chance that if the driver were paying attention, she would have seen the pedestrian before the car did.

The thing about safety systems and standards are that there should be some redundancy in them. People make mistakes, so you shouldn't have to rely on every actor acting perfectly to remain safe. In this case, there was a failure on many parts leading to a tragedy.

It's definitely worth investigating whether Uber had done enough to ensure that operators will be staying alert during the entire test, and whether Uber had done enough testing before being on open roads of small, possibly moving obstacles on dark parts of the road, like pedestrians, deer, and so on.

I'd also put a good amount of blame on whatever department designed that portion of road and sidewalk.

Re:Uber disabled Volvo SUV's safety system before fatality by ealexhudson on Tuesday March 27, 2018 @03:08AM (#16686013)
I don't think it's weird. I called this a few days:

https://news.ycombinator.com/item?id=16658566

Having them disable a standard feature isn't per se a problem. However, it is an issue if the car is demonstrably less safe with it removed. It's possible that this car was going too fast for this feature to engage anyway (depends on the age of the XC90: if it was post-2014, it should have kicked in).

The Volvo system uses a LIDAR just like the main Uber system. In principle they should have been able to replicate this safety feature pretty much precisely.

The question then arises; if they're making modifications on systems like that, what other modifications have they made? Are the modifications correct?

The issue with the headlamp angle implies there could be other problems. Modern cars are incredibly complex; pulling out single features might be OK when they are add-ons, but the XC90 ships with a lot of this stuff in every model.

Re:Uber disabled Volvo SUV's safety system before fatality by OrganicMSG on Monday March 26, 2018 @11:40PM (#16685022)
>having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable.

Multiple redundant systems that do not need to talk to each other to coordinate is generally safest.

And if the existing systems can not be coordinated with and can make cars less safe because of this, then how are drivers managing it?

Re:Uber disabled Volvo SUV's safety system before fatality by pwthornton on Tuesday March 27, 2018 @10:32AM (#16688758)
It's not weird at all.

The point is that Uber should only be disabling these safety features on public roads if their technology is superior.

If their technology is less safe, and in this case it appears to be that it is, they shouldn't be driving around on public roads. Keep it to test tracks.

Volvo's system would have either stopped or slowed enough to make this crash non-fatal. That's worth pointing out. There is no point in Uber testing technology that is worse than what you can buy any number of cars today.

Re:Uber disabled Volvo SUV's safety system before fatality by FussyZeus on Tuesday March 27, 2018 @10:48AM (#16688943)
...playing Devil's Advocate, as GP said, it's common practice when testing any self driving tech to disable these features on the cars in question so your automated system isn't fighting with another set of automated systems trying to control the same vehicle. That sounds far more unsafe and unpredictable than just disabling the built in ones.

Personally I think it's naive and unreasonable to think we're going to get totally safe and rock solid self-driving tech without at least a few coffins worth of people who gave their lives to show us how it could go wrong. There isn't one safety system in existence that wasn't built on top of a pile of corpses.

Re:Uber disabled Volvo SUV's safety system before fatality by polack on Tuesday March 27, 2018 @11:05AM (#16689150)
I've had a car with automatic breaking system for a while now and my experience is that they do not kick in easily. Had to test it with a friend that throw an inflatable mattress in front of the car to make it react.

It should not be too hard to make these systems co-exist. If one of the systems says break then the car will break. The few times the car breaks when it really didn't need to should be an acceptable side-effect during testing.

Re:Uber disabled Volvo SUV's safety system before fatality by FussyZeus on Tuesday March 27, 2018 @11:31AM (#16689450)
Except this is a prototype, not a final product. Were a self driving car being made for public use, there would not be a pre-existing safety system to bypass or modify; it would be built in, or at the very least, the car would be delivered for modification with that system already removed.

If you had a firewall in your network, for example, that had both a firewall system that you configured, and alongside it, a firewall configured by a third party with "best practices" in mind, that would be frustrating to configure and test because you'd never know for sure if it was a configuration you didn't setup correctly, or if it was this other firewall that you couldn't operate interfering with whatever it was you were doing.

When designing things like this, step 1 is you remove variables that could change the behavior of the device in question without you knowing why it did. I'm sure once completed, a self driving car system could well be made to interface with the safety systems otherwise present, or could be designed in such a way where they wouldn't conflict. When you're in the thick of building that system, however, an additional (and by your comment, not exactly predictable) system to contend with, that you have no access to, is going to be a pain in the rear.

Re:Uber disabled Volvo SUV's safety system before fatality by polack on Tuesday March 27, 2018 @11:42AM (#16689570)
No, you do NOT remove security systems when trying out an untested system. If anything you should add MORE.

Again, the stock auto break systems rarely engage unless there is real danger. By that I mean last second breaking, absolute last second. Most people can drive a lifetime without ever having them engaged. If you cant program a self driving car that can handle that then you're in the wrong business.

Re:Uber disabled Volvo SUV's safety system before fatality by tzakrajs on Tuesday March 27, 2018 @11:21AM (#16689328)
Exactly. People are acting like these self-driving programs are so fragile that they can't handle a second system rarely taking control.
Re:Uber disabled Volvo SUV's safety system before fatality by amluto on Tuesday March 27, 2018 @11:01AM (#16689095)
> Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable.

If I had a Volvo XC90, I would want to operate it with two systems: the collision avoidance features plus an advanced system of muscles, eyeballs and brain. The whole point of collision avoidance is to operate in conjunction with a driver. Self-driving should be more or less just another driver.

Re:Uber disabled Volvo SUV's safety system before fatality by lmilcin on Tuesday March 27, 2018 @04:18AM (#16686387)
The problem is that the standard, "dumb", security system is still more effective that "self"-driving solution from Uber.

How is a "self"-driving solution allowed to drive in real traffic when it can't break in a simple situation where relatively simple, existing systems can already reliably break.

Re:Uber disabled Volvo SUV's safety system before fatality by chronolitus on Tuesday March 27, 2018 @07:37AM (#16687318)
s:break:brake:g
Re:Uber disabled Volvo SUV's safety system before fatality by lmilcin on Wednesday March 28, 2018 @06:49AM (#16696756)
Thank you! I am Polish living in Poland and still learning.
Re:Uber disabled Volvo SUV's safety system before fatality by gambiting on Tuesday March 27, 2018 @04:02AM (#16686318)
>>having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe.

I think this case clearly demonstrates otherwise. Think of the factory fitted as fail-safe - if their own autonomous tech fails(like it did here) you have the one last system that will prevent collision(and it looks almost certain that it would have done that here).

Re:Uber disabled Volvo SUV's safety system before fatality by couchand on Tuesday March 27, 2018 @01:23AM (#16685497)
> having a second system running that can't coordinate with the self-driving system

It seems like "not running into stuff" is about the only level of coordination necessary here.

Re:Uber disabled Volvo SUV's safety system before fatality by matthewdgreen on Tuesday March 27, 2018 @07:02AM (#16687120)
I have an XC90. The collision avoidance system has been helpful in a couple of instances (for sudden car stops, not pedestrians). However it also “misfires” a few times per month. Typically this occurs when I’m driving on a curving stretch of road with cars parked along the side: due to the curve, the system will sometimes interpret the vehicles on the side of the road as being obstacles in my path. When it does, I get an alarm and sudden unexpected braking. The first time(s) this happened I was genuinely terrified.

I have no idea how this input would be interpreted by an autonomous driving system. However if I was designing such a system I would be pretty concerned about the potential safety impact. At very least I would want to add code to recognize these special braking events and then I would need to test this code... which of course would be impossible to do in simulation because Volvo’s system is probably closed source and likely exists as a hardware black box. At the end of the day I would probably be more comfortable designing my own collision avoidance system than relying on one that I couldn’t test easily, and that might spit unexpected inputs into my already-complex system.

So no, I don’t see this is a simple question of “adding redundancy”. I think what Uber did here probably makes good sense. The fact that their own collision-avoidance systems didn’t fire is, however, a problem.

Re:Uber disabled Volvo SUV's safety system before fatality by icc97 on Tuesday March 27, 2018 @08:22AM (#16687573)
As has been said elsewhere false positives of stopping suddenly are bad, but are better than false negatives of not stopping when there is something there.

Also the collision-detection systems should be getting tested so that they improve too. You want your fall back systems to be more reliable than the regular system. So if you turn off the collision-detection systems never fix the false positives and false negatives then you'll always have these problems.

Re:Uber disabled Volvo SUV's safety system before fatality by cptskippy on Tuesday March 27, 2018 @10:37AM (#16688806)
I would argue that AI systems should be able to handle sudden and unexpected events like system malfunction or failure.

> At very least I would want to add code to recognize these special braking events and then I would need to test this code... which of course would be impossible to do in simulation

Why? Does Volvos system raise events differently in context? I would imagine it's pretty simple to simulate an event by driving the car into a wall and capturing a dump of the CAN bus.

The autonomous system could take the Volvo system firing as an event like anything else that it needs to reconcile in the system. If anything the autonomous system could be used to improve Volvo's system by capturing extensive data dumps of the events leading up to misfire events.

Re:Uber disabled Volvo SUV's safety system before fatality by jacquesm on Tuesday March 27, 2018 @07:47AM (#16687390)
You should file a bug on that. Someone coming up behind you might end up getting hurt due to the chain reaction of your car braking unexpectedly (and without a good reason to do so). There's a typical domino effect from one car braking where the reaction times compound and cars further back end up having to brake harder and harder to avoid a collision until one ends up being out of room and out of time.
Re:Uber disabled Volvo SUV's safety system before fatality by Piskvorrr on Tuesday March 27, 2018 @05:29AM (#16686690)
That's a goal which is easy to formulate, but hard to put into code.
Re:Uber disabled Volvo SUV's safety system before fatality by JorgeGT on Tuesday March 27, 2018 @05:51AM (#16686801)
The human driver can't coordinate with the self-driving system either, he just makes overriding inputs. Treat the built-in safety system as another overriding actor.
Re:Uber disabled Volvo SUV's safety system before fatality by couchand on Tuesday March 27, 2018 @12:10PM (#16689880)
Clearly, since they failed so miserably here.

But -- here's the crux of the thing -- it's not a new goal, it's the same goal they (should) already have.

Re:Uber disabled Volvo SUV's safety system before fatality by iainmerrick on Tuesday March 27, 2018 @05:48AM (#16686791)
The code exists! The collision detection and emergency braking feature which many new cars already have, and which (I assume and hope) has been heavily road-tested.
Re:Uber disabled Volvo SUV's safety system before fatality by Piskvorrr on Tuesday March 27, 2018 @12:02PM (#16689787)
That's what has been said of Uber's code as well. Hope-based safety doesn't seem to be working.
Re:Uber disabled Volvo SUV's safety system before fatality by iainmerrick on Wednesday March 28, 2018 @03:40AM (#16696110)
But my point is, Uber’s self-driving car has lots of goals -- drive safely, drive legally, drive quickly, drive comfortably, go in the right direction.

A low-level safety system has a much simpler requirement: if it looks like the car is about to hit something, hit the brakes.

And this second category of code is something that already exists, deployed in production today by multiple car manufacturers.

That’s the bit Uber turned off. “It’s a hard problem!” doesn’t seem like a good explanation.

Re:Uber disabled Volvo SUV's safety system before fatality by Fomite on Tuesday March 27, 2018 @09:01AM (#16687850)
I can see it from the perspective of Volvo pushing this - one of their cars is now widely shown in the media killing someone, and their whole reputation is safety. I'd also want people to know that Uber was tinkering with the standard safety features, even if it was for good reason.
Re:Uber disabled Volvo SUV's safety system before fatality by iainmerrick on Tuesday March 27, 2018 @05:44AM (#16686768)
If there's a human driving the car, it's safer to have both the human driver and the automatic emergency stop system.

Exactly the same deal if there's an AI driver.

Re:Uber disabled Volvo SUV's safety system before fatality by raverbashing on Tuesday March 27, 2018 @08:50AM (#16687767)
While it could make it less safe in very specific circumstances, note, it's the same as with a human driver

The system kicks in when there is a high probability of a collision ahead. Think of it as the instructor in the driving school that slams the brakes when you're doing something wrong.

So basically, the driver (be either human or another system) did something wrong already if that system kicks in

Re:Uber disabled Volvo SUV's safety system before fatality by cptskippy on Tuesday March 27, 2018 @10:16AM (#16688572)
Most industrial controls have safety interlocks which are independent systems with hard limits. Why is it unrealistic to think that an AI system couldn't be trained to work with-in the limits imposed by these safety systems? If the AI system is being interrupted by the independent safety systems in the car the the AI is doing something wrong.
Re:Uber disabled Volvo SUV's safety system before fatality by fixermark on Tuesday March 27, 2018 @11:28AM (#16689420)
It reads like a story press-released by Volvo to make sure Volvo is CYA'd in this in the public eye (lest uninformed consumers find themselves wondering how good Volvo's collision avoidance is if it couldn't stop that Volvo-branded car from hitting that woman in Arizona).
Re:Uber disabled Volvo SUV's safety system before fatality by Ntrails on Tuesday March 27, 2018 @08:18AM (#16687551)
> This makes sense: presumably there's no good way to talk to the built-in system; having a second system running that can't coordinate with the self-driving system would likely make the combined system less safe. Even if communication between the two were possible, I'd still imagine that just having a single system operating the care would be much more desirable.

That's the exact argument I make for never buying a car with that technology because I'm driving it and I have no way to communicate with it...

I'd be super happy to be driven around in an autonomous car, but have zero interest in auto-lane-keeping-pilot-collision avoidance-parking etc

Re:Uber disabled Volvo SUV's safety system before fatality by tzakrajs on Tuesday March 27, 2018 @11:16AM (#16689289)
Aren't humans glorified self-driving systems? The second system is a backup for when the human makes a terrible mistake, similar to how this could've been a backup for the computerized self-driving system.
Re:Uber disabled Volvo SUV's safety system before fatality by eklavyaa on Tuesday March 27, 2018 @03:53AM (#16686279)
I wonder why they bought volvo's when they don't wanted to use there radar system or were not able to use the radar system. And is it that easy to disable such an important feature which Volvo brag about ?
Re:Uber disabled Volvo SUV's safety system before fatality by leetcrew on Tuesday March 27, 2018 @08:40AM (#16687688)
imagine i'm considering buying 24,000 nice cars from you, but i need to be able to disable certain systems. i feel like you would help me out.
Re:Uber disabled Volvo SUV's safety system before fatality by eklavyaa on Wednesday March 28, 2018 @01:48AM (#16695661)
but if I don't want that feature then my need of disabling is superficial
Re:Uber disabled Volvo SUV's safety system before fatality by HenryBemis on Tuesday March 27, 2018 @05:12AM (#16686627)
It would make sense that some Volvo's systems were disabled, only to be replaced by Uber's central Computer (or whatever is called).

This way there won't be 2 breaking systems competing with one-another. Different systems controlling the same end-points is never a good idea.

Re:Uber disabled Volvo SUV's safety system before fatality by JorgeGT on Tuesday March 27, 2018 @06:13AM (#16686898)
> there won't be 2 breaking systems competing with one-another. Different systems controlling the same end-points is never a good idea.

Aerospace engineer here. Different systems controlling the same end-points is literally a basic engineering safety technique. Not only you have the human pilot as a competing system controlling the same end-points as the autopilot, you may have even different competing autopilots.

I had a professor at the Uni who was part of the engineering team of the flight computer of a certain jet fighter. Two engineering teams were given the specs of the system and told to produce a flight computer. The two teams weren't allowed to talk to each other to prevent bugs cross-pollinating the two systems. After all, what good is to have two identical system for redundancy if they can be affected by the same bug? (That's also why pilots eat different meals in-flight!)

Then, the two completely different systems are installed, duplicated, into the plane and put to the test. The four computers must perform equally as per the specs. If the recommended output of a single computer diverges from the other three, that computer is faulty and disregarded. If two computers of the same design have a bug and start giving out-of-envelope readings but the other two are consistent, control is handled to them.

I'm certainly very appalled to see in this thread how safety engineering techniques that every engineer learns in University is completely disregarded or even reversed. At least in the aerospace community, development and implementation of these safety techniques has been paid in blood. These are lessons that shouldn't be forgotten when developing other transport technologies IMHO.

Re:Uber disabled Volvo SUV's safety system before fatality by jacquesm on Tuesday March 27, 2018 @07:51AM (#16687406)
Software people as a rule absolutely suck at dealing with exceptions and in general tend to make a ton of assumptions about the underlying hardware and sensors that may not hold up in practice. The 'happy path' is where 90% of the attention goes, when actually 90% of the attention should go to the exceptional bits. But as an industry we're slowly learning and the habits and practices of other engineering disciplines are slowly seeping into to software.

Aerospace (and medical) is where things are much better.

Re:Uber disabled Volvo SUV's safety system before fatality by jschwartzi on Tuesday March 27, 2018 @06:40AM (#16687004)
Most of the commenters did not study engineering. They studied computer science. So the only exposure they would have had is working with someone like you.
Re:Uber disabled Volvo SUV's safety system before fatality by JorgeGT on Tuesday March 27, 2018 @07:38AM (#16687320)
I wonder if those in self-driving development at Uber et al. share the same non-engineering background. I think it would be ultimately in the interest of these companies that stringent regulation is enacted, so the public can feel as safe and confident riding self-driven vehicles as they feel riding a thin aluminium tube with two gigantic kerosene hairdryers flying at 35,000 ft.
Re:Uber disabled Volvo SUV's safety system before fatality by vegardx on Tuesday March 27, 2018 @05:45AM (#16686775)
Except for emergency systems. If you have two or more independent systems that are able to stop, should any of them should detect an anomaly, you are far more likely to survive a partial system failure. The end result of all systems identifying the problem is the same.

In a modern car there are plenty of systems that independently will start. An example is some cars that are fixed to prepare your breaks if you take your foot off the accelerator quickly, more or less giving you instant breaking. At the same time, completely independent of this, emergency breaking might already have been started by the collision detection system.