The irony is that this is 100% the what MobilEye said in their in their model demo I think 3-4 years ago. Their CEO said that the regulators cannot regulate all breaking conditions and should should only test for false negatives exactly to prevent this. He stated that dealing with flase positives is going to be in the best interest of every manufacturer and the easiet thing to do is to disable the breaks completely but that would be fairly easy to detect in any structured test or a test drive. Any thing else would be a problem because you would be stacking tolerances and worse creating test cases with a clear conflict of interest where the interests of the car maker and the regulator align.

This comment made my brain hurt. I think you have something interesting here, that’s why I read it four times and still didn’t understand.

Please re-write this for clarity.

Mostly the message is that regulators shouldn't be investing their resources in checking if the car breaks too often (from false positive in obstacle detection) because the car company has strong incentive to reduce unnessary breaking or driving would be unpleasant and slow for a large portion of the drivers.

The regulators should only test for false negatives, where the car should have stopped, but did not detect the obstacle (false negative), because there, it is a clear threat to safety, and the car company's incentive, while definitely still present, is less pure, as the amount of false negatives is a direct trade off with the quantity of false positives (because it is a treshold: a minimum confidence level from which you decide that there is indeed something in front of the car and you need to break), which make driving more awkward for 99% of drivers

Does any regulator actually give a damn about a self-driving car braking too often?

The persistent use of "break" instead of "brake" is difficult for me in this specific context.

This comment page has dozens of this misspelling. I'm used to it on reddit, I thought the HN crowd is more intelligent than reddit...

The TL;DR as I read were

1. Regulators want aggressive breaking but carmakers want smoother driving

2. Manually tuning all the edge cases for cases where the software is uncertain what's happening will lead to fragile monolith black boxes

Aggressive random braking is also a safety issue.

Yep, this is the thing that worries me the most on some of these systems right now. They are going to get rear-ended a lot for the next few years, IMO.

Rear ends have basically no fatality rate though. They do have material costs but if optimizing for no loss of human life it sounds more appealing.

These kind of tradeoffs were things every self-driving car software developer KNEW they were going to have to deal with - the most extreme being the one where the software has to decide who to kill and who to save:

https://www.theglobeandmail.com/globe-drive/culture/technolo...

And rules out use on snowy roads.

Why? Brakes works fine on snowy roads. Distances can be adjusted for snowy roads. It happens every winter where it snows a lot. I don't see a problem.

Not sudden random braking, no. The most important advice for driving on slippery surfaces is to avoid sudden braking, and in general to be careful when you brake.

That'll teach people to keep their damn distances!

I second the motion.

> If you can't give your "self-driving software" full access to the brakes because it becomes an "erratic driver"

You misunderstood. The Uber software had sole control of the brakes (plus the human of course). The Volvo factory system was disabled so that it didn’t have negative interaction with the Uber system.

Your mistake is understandable. The article was poorly written, perhaps due to a rush to publish, as is the norm these days. Even if the NTSB report was unclear, that doesn’t excuse clumsy reporting.

If you’ve ever done significant mileage in a car with an emergency braking system you probably have experienced seemingly random braking events. The systems favor false positives over false negatives.

Is that true? A cousin comment appears to indicate Uber had a parallel emergency breaking system and that is what failed to trigger:

> At 1.3 seconds before impact, the self-driving system determined [...]

Unless this is exceptionally poorly worded, it certainly sounds like self-driving system was the one doing the determination.

https://news.ycombinator.com/item?id=17144741

I wrote it carefully and I stand by it. If it can't deal with it because your system is getting too confused for any reason, you don't have a self-driving system. Being able to function enough like a human that the safety features on the car don't produce an unacceptable result is a bare minimum requirement to have a self-driving car.

This isn't horseshoes, as the old saying goes. I unapologetically have a high bar here.

Far as I can tell, the done thing in the industry is to disable all other safety systems (or to not have any in the first place) and to delegate safety entirely to the self-driving AI.

The charitable interpretation of this is that the industry believes that self-driving AI is a safety feature of greater quality than, say, lane assist or auto-breaking.

The less charitable one is that they find it too much work to integrate their AI to other safety systems. Which, to be fair, is really going to be a lot of extra work, on top of developping self-driving.

>You misunderstood. The Uber software had sole control of the brakes (plus the human of course). The Volvo factory system was disabled so that it didn’t have negative interaction with the Uber system.

That's not how I read it, or how any of the journalists who are reporting the story are reading it. Uber disabled the self driving software's ability to do an emergency stop when it detected it was going to crash. The Volvo system is separate and also was disabled when the car was in self driving mode.

https://www.bloomberg.com/news/articles/2018-05-24/uber-self...

>Sensors on an Uber SUV being tested in Tempe detected the woman, who was crossing a street at night outside a crosswalk, eventually concluding “an emergency braking maneuver was needed to mitigate a collision,” the National Transportation Safety Board said in a preliminary report released Thursday.

>But the system couldn’t activate the brakes, the NTSB said.

That's the only reading that makes sense to me, otherwise why did the car fail to attempt to stop when it detected the pedestrian and knew it was going to hit them?

This is from the NTSB report:

"At 1.3 seconds before impact, the self-driving system determined that an emergency braking maneuver was needed to mitigate a collision (see figure 2). According to Uber, emergency braking maneuvers are not enabled while the vehicle is under computer control, to reduce the potential for erratic vehicle behavior."

I believe you are the one who has misunderstood.

I think you're wrong. The Uber software controls the brakes, thru whatever means they're controlling the car. It has to – the car has to stop somehow!

"emergency braking maneuvers" refers to an additional automated (software) system for automatically applying the brakes in an emergency (that's detected by that additional system).

No, see footnote 2 of the NTSB report:

>2 In Uber’s self-driving system, an emergency brake maneuver refers to a deceleration greater than 6.5 meters per second squared (m/s^2).

So, Uber's self-driving system has command to brake normally, but it cannot "slam" the brakes. The other system from Volvo's is deactivated when Uber's is working and cannot brake at all. Thus, since Volvo's is deactivated and Uber's won't brake if it judges that a deceleration of >6.5 m/s^2 is needed, it turns out that in automated mode, the car actually lacks the ability to trigger emergency braking at all, hoping instead that the driver will somehow notice. But in a sadistic twist, no warning is given to the driver at any moment that they need to slam the brakes.

At 1.3 seconds, and with as many false-positives as it sounds like the system has, I don't think a warning would have helped.

No, it would probably take a large part of that time to even react, especially if you are looking at a screen.

However, if the safety driver was trained to brake immediately upon warnings it could have worked quite well. But that would negate the removal of e-brake actuation.....

If the car cannot use emergency braking it could at least decelerate. It looks like some hack where the engineers just commented out the code for brakes and left decision to the driver.

edit: nevermind, see makomk's response to me.

The NTSB report is pretty unclear, I had to re-read it several times and I think you're correct that the "emergency braking maneuvers" they refer to are the Volvo ones. It's strange though that they word it as

> At 1.3 seconds before impact, the self-driving system determined that an emergency braking maneuver was needed to mitigate a collision

Is the Uber self-driving system able to interact with the Volvo system? Or are they calling the Volvo safety features a second "self-driving system"? And what were the steps, Uber realizes it needs an emergency braking maneuver and sends a signal to the Volvo system which then responds with "I'm disabled" ?

I understand why the Volvo features may be disabled, but it's alarming that the self-driving system made no attempt to brake at all when it was fully "aware" it would hit someone.

The report does mention the Volvo braking features by name a few paragraphs earlier though... So I'm still not entirely sure.

They definitely seem to be referring to the Uber self-driving system. The sentence "At 1.3 seconds before impact, the self-driving system determined that an emergency braking maneuver was needed to mitigate a collision (see figure 2)." has this footnote attached to it: "In Uber’s self-driving system, an emergency brake maneuver refers to a deceleration greater than 6.5 meters per second squared (m/s^2)".

Thanks, don't know how I missed that, I'll blame my phone screen haha.

> The vehicle operator is relied on to intervene and take action. The system is not designed to alert the operator.

If what you said is true, then the vehicle operator would not be relied on to intervene because the Uber self-driving software would apply the brakes. Since the vehicle operator is relied on to intervene, this indicates the Uber self-driving software has its emergency braking disabled.

> If you’ve ever done significant mileage in a car with an emergency braking system you probably have experienced seemingly random braking events. The systems favor false positives over false negatives.

This is not my experience with VW's 2016-model-year system.

Soemtimes it stops a second or two before I would've. I haven't had any false positives, though.

Same here in our 2016 wv, we had a beep once for a warning in an intersection with traffic nearby, but never a false positive.

I presume that expectation was that the human driver was suppose to be fully engaged to take evasive actions if and when required. What jumps up at me if that fact that the system does not alert the driver when his/her intervention is required.

I'm not sure it matters, because going from disengaged to fully engaged and actually breaking to a stop in 1.3s is not happening for an average human. By the time the system alerts the driver, the ped is already dead.

But an attentive human would have noticed the pedestrian around the same time the system was confused, ~6s before impact. Enough time to swerve and stop.

At 6 sec before impact the driver for sure would assume the system will notice and brake in time. Until the driver realizes that the automated system will NOT brake its very likely too late. Automatic systems which rely on humans braking in time have no place on the street, imho. This may be different for lane assist where malfunctioning is more obvious and leaves more time for intervention. Although even in this case the latest Tesla accidents may tell a different story.

> If the self-driving software is not fully capable of replacing the driver in the car you have placed it in, ...,you do not have a safe driving system.

Nobody disagrees with you and that is explicitly the reason why a human is on board, so I am not sure what you are arguing against.

Is that enough?

I think it's very clear that a human driving a normal vehicle is different from a human sitting at the wheel of a self-driving (semi-self-driving?) vehicle. You simply cannot expect a human to remain as engaged and attentive in such a passive situation.

It's baffling to me that they chose to deactivate emergency braking without substituting a driver alert. If the false-positives are so frequent as to render the alert useless (i.e. it's going off all the time and you ignore it) I don't think these vehicles are suitable for on-road testing.

You can lean fairly heavily on the 'still dangerous, but better' argument in the face of 40,000 US vehicle fatalities each year, but there are limits.

Except it’s patently clear that is a flawed concept. Here’s a prime example where a human, not paying attention because the car has been successfully driving for a while, is given 1-2 seconds to emergency stop the car. That is not enough time to process what’s going on and take over. Even 10 seconds is likely not enough.

Worse, under Uber's regime the one human had to deal with both emergency situations (without warnings from the system) and instrumentation feedback (without a HUD), so the intended operation of this test was that the safety driver spends half the time looking down and away from the road.

And they also ran tests at night. This is not an obstacle for the LIDAR but is an extra challenge for the human.

Yeah, and it is also why they had a driver facing camera to insure that driver's were taking this responsibility seriously.

It seems that Uber wasn't actually verifying that, though.

The driver was also in charge of monitoring and classifying/tagging instruments messages appearing on monitors on the center stack.

I doubt the camera can give you a good idea of whether or not the 'driver' starts to daydream. They could be looking at the road and not paying attention. Worse, the 'driver' could be unaware they aren't paying attention, rendering their intention to stay alert moot.

Of course that is possible, but in this case the driver was clearly not looking ahead. I guess it is possible that this was the first time the driver did that, but I think that is unlikely.

Maybe he was just making a statement.

So, let the two software systems fight with each other during emergencies and hope it somehow results in something positive?

That sounds like a terrible idea.

On the contrary: the Volvo system is designed not to fight but to override any other input, including the human driver. In any other Volvo car currently on the road that detects an obstacle, the Volvo system will override the driver and bring the car to a stop even if the driver insists in hitting the obstacle: https://www.youtube.com/watch?v=oKoFalJiazQ

Independently-working override-capable systems are the base of engineering redundancy safety. See airborne collision avoidance systems (ACAS), which will automatically and forcefully steer an aircraft to avoid a collision if necessary: https://en.wikipedia.org/wiki/Airborne_collision_avoidance_s...

I don't have experience with self-driving cars, Uber or Volvo safety systems, but my Jeep's backup collision prevention often slams the brakes if I'm backing up and the camera detects potholes, oil stains, or irregular painted lines in the parking lot.

If any of the systems are vulnerable to any such false positives, and equipped to enable emergency braking to avoid them, even on the highway, it's not hard to imagine why they might be disabled, especially during testing phases.

I think it's fair to say that at this early stage in product development, there's probably no 'always right' answer for how to handle a given obstacle without considering locality, driving speed, road conditions, likelihood of false positives and negatives, etc.

They shouldn't be testing on public roads and highways then.

Yeah mine does that too. I wish they had a sensitivity option like they do with the forward collision sensors.

Hey there's an idea for Uber, maybe instead of disabling the forward collision system entirely they could just decrease the sensitivity to lessen false positives (like in our Jeeps)?

TCAS systems won't automatically take over. Instead they audibly issue an order to the flight crew, describing the action that must be taken, and the flight crew must take the action.

Instructions from TCAS are near absolute in their priority. If ATC says to do something different, you ignore ATC and do what TCAS says. If the pilot in command says to do something different, you ignore the pilot in command and do what TCAS says. If somehow God Himself is on your plane and tells you to do something different, you ignore Him and do what TCAS says. Compliance with TCAS is non-negotiable, and the Überlingen disaster[1] is the bloody example of why it's that way.

Self-driving/autonomous-car systems need to have a similar absolute authority built in. If Uber disabled theirs because of false positives, it's a sign Uber shouldn't be running those cars on public roads.

[1] https://en.wikipedia.org/wiki/2002_%C3%9Cberlingen_mid-air_c...

AFAIK Airbus has a TCAS-capable AP/FD, certified at least for the A380, which can automatically fly RAs by itself: https://www.skybrary.aero/index.php/Autopilot/Flight_Directo...

If the Uber engineers shared your confidence that it would play out that way, they would not have disabled it.

Redundant safety systems are a great idea, evidently Uber needed more of them, and integrating with the Volvo system might have been a reasonable option. It's silly to suggest that the integration would necessarily have been trivial, though. That's what I'm objecting to.

A woman is dead because of some combination of overconfidence, hubris, and ignorance of Uber’s self driving engineers.

The above traits aren’t exclusive to Uber’s “engineers”, but are lethal when applied to the engineering of life safety systems.

I am not sure if I would have been able to avoid such a situation myself. That's the only thing that keeps me from excoriating Uber...

How many people are still alive because Uber had the restraint to not slap control systems together willy-nilly, like so many people here seem to think is an obviously great idea?

A lot more than one, that's for sure.

Yikes. I would not allow a business to continue to exist whose argument is “at least we’re only partially wreckless”.

See how that holds up in civil court in front of a jury, or any legislative body Uber might need to convince to allow their operation in a jurisdiction in the future.

“Just think of how many more people we would’ve killed if we didn’t care at all!”

"at least we’re only partially wreckless" has been Tesla's standard operating procedure for a while now. I'm guessing you think Tesla shouldn't exist either.

Oh, they'll get crucified for it, I'm sure -- because the public doesn't understand that integration is never trivial, and that "obvious" integrations aren't always good ideas.

I had thought an audience of developers would "get it," since we deal with fallout from ill-conceived integrations every day, although admittedly in a far less spectacular form than control system engineers.

Unfortunately, the uber hate train has already left the station and there's no slowing it down until the investigation finds (or doesn't) actual evidence of negligence rather than clickbait guesswork by armchair engineers. Too bad.

> I had thought an audience of developers would "get it," since we deal with fallout from ill-conceived integrations every day, although admittedly in a far less spectacular form than control system engineers.

You do a disservice to the audience by assuming it would be understanding of grossly negligent behavior.

Failures happen, that is to be expected. If you're building self-driving vehicles, you're supposed to be engineering for those failures. Disabling two life safety systems (Volvo's AEB and Uber's own AEB) and relying on a single inattentive human driver? I don't understand how that's understandable or justifiable in any scenario besides a carefully controlled test track.

I think you’re misunderstanding. According to the report, it’s not just the Volvo emergency braking system that was disabled. Uber’s self-driving system had its own emergency braking feature that was also disabled.

How many people were not killed by Uber? That's a terrible way to look at things. It is better to wonder _why_ Uber killed a person. The answer appears to be right in the NTSB report about the brake system.

You're being very charitable in your assumption that Uber 'engineers' gave it much thought at all.

(Were any Professional Engineers even involved in this? Or was it just a bunch of "software engineers"?)

However much thought they gave it, I guarantee that everyone in this comment train has given it much less. It wasn't a trivial decision, their choice is not prima facie evidence of negligence.

Again, very charitable. Your assumptions are baseless, seemingly motivated by little more than your trust that incredible incompetence and negligence is rare in the real world.

Again, not at all charitable. Your assumptions are baseless, seemingly motivated by little more than your mistrust that in the competence of people that have been actively working on this problem, many for years.

He said that he can "guarantee" Uber engineers gave it careful consideration (or at least more than people in this thread), but in truth he can do nothing of the sort. There are a plethora of examples where engineers and "engineers" didn't give a problem careful consideration and it got people killed. There is no rational basis for guaranteeing this is not one of them.

There is no "guarantee" that uber "engineers"/engineers put more thought into this than "their system is inconvienent, tear it out" or "we don't need their system because ours will be better, tear it out." Nobody can guarantee Uber engineers were not stupendously negligent until the investigation is complete. Anybody who thinks they can guarantee that has an irrational basis for thinking they can provide such a guarantee.

You might be troubled to know that the guidance control systems on modern spacecraft and many aeronautics subsystems work this way -- at least three redundant systems have access to sensor data and each make independent decisions about the course of action, if any, needed. In short, they vote, and majority wins.

There is no "fight with each other". If any system says "break", the car breaks. If none says that, the Uber system has control.

Every time you drive a modern car, you fight for control with a computer driving system.

Somehow, it works out.

Unclear why this is being downvoted. Modern cars have all sorts of computer driving systems, e.g. ABS. If you learned to drive without it, you still think to pump even though the computer does a better job if you just slam on the brakes.

The point is that so far most (all?) of these computer systems don't have a failure mode where it kills pedestrians, because their scope is very limited.

Because one doesn't "fight" with ABS not wanting to break. It will break when you hit the brake.

Same with power steering: it won't steer into a different directorion. You don't have to "fight" with it.