> We need to avoid treating deaths as progress in the pursuit of better things

Au contraire. Go read building codes some time. There's a saying that they're "written in blood" - every bit, no matter how obvious or arbitrary seeming, was earned through some real-world failure.

The death itself isn't progress, of course. But we owe it to the person to who died to learn from what happened.

Federal Aviation Administration's Joint Order 7110.65 is exactly what you are talking about. It is the manual that Air Traffic Controllers live by. To include situations that involve diverging aircraft or taxiing instructions and how to handle said situations. The entire manual was practically written from real-world experience.

This reminds me of the story behind the iron ring and the collapse of the second narrows bridge in Vancouver, BC.

http://www2.ensc.sfu.ca/undergrad/euss/enscquire/Vol10No2/pa...

It used to be the case that those “codes” were also written in the blood of the builders – as per the Code of Hammurabi.

You might enjoy _The History of Fire Escapes: How to Fail_, a talk Tanya Reilly gave at a DevOps Days conference earlier this year.

https://www.youtube.com/watch?v=02KEKtc-5Dc

Not just building codes, but a lot of car safety regulations and practices too. Stuff like crumple zones, ABS, breakaway road signs, safety glass, etc.

You seriously think Uber wouldn't try to claim "commercial in confidence" or "trade secrets" rights to all the data from every single death?

Clearly I think Uber is a benevolent entity that has all of our best interests at heart. Also, I eat babies.

Or, you know, don't jump on comments for their not explicitly addressing the hobby horse you're riding. Frankly I just wanted to express a better engineering context around the loss of life without getting into the political bullshit for once.

>There seriously fucking needs to be a consortium for exactly this purpose: sharing system failures.

This implies they're using the same systems and underlying models. If one model hit a pedestrian because of a weakness in training data plus a sub-optimal model hyperparameter, and therefore was classified a pedestrian in that specific pose as some trash on the street, how do you share that conclusion with other companies models?

I guess it depends on the data available, but I don't think the hyperparameters are what's needed. You just need to see what the car sensed and how it responded. Then the other car companies can try to replicate the similar circumstances and see what their models would do.

> You just need to see what the car sensed

I don't think all self-driving cars have the same sensors.

If you have LIDAR model X, and they were using LIDAR model Y, will your system "magically figure it out?

If your car has cameras at 5ft high, and the data is from a camera 6ft high?

Sure, someone could release the data, but will it screw up your models more than it fixes them?

(I totally agree the data should be released, I'm just not sure other self-driving cars will directly benefit. Certainly they can indirectly benefit from it.)

Is the design of self driving cars so limited that we will have to go through this every time carmakers must redesign hardware and/or vehicles themselves? Will the experiences of a sedan be impossible to transport to the experience of a semi truck? And vice versa?

The idea you present is possible, but I have to wonder how viable it makes the idea of self driving cars.

when self driving cars becOme ubiquitous it will be thru a renting model and therefore the fleet can be updated accordingly by companies owning them.

I understand, but the error may be in how it interpreted what it sensed. This is callous language, but if the models interpreted the pedestrian as trash on the street, then how it responded (driving over it) is not inappropriate.

> I understand, but the error may be in how it interpreted what it sensed.

It may, it also might not. If sharing data fails, further methods would be needed - but it's a good start for figuring out what data should be recorded for comparison. If the data is entirely incompatible, then we should have regulation to require companies to at minimum transcribe the data into a consumable format after an event such as this.

> ... if the models interpreted the pedestrian as trash on the street, then how it responded (driving over it) is not inappropriate.

If the models saw anything at all, they should not have driven over it. Even human drivers are taught that driving over road debris is dangerous. At minimum it puts the car/occupants at risk - in extreme cases, the driver may not recognize what it is they are driving over.

If this isn't a case where the car was physically unable to stop - it's more likely the telemetry didn't identify the person as an obstacle to avoid at all.

The idea is to come up with a certification test suite, and the cars have to pass it. Add/modify the tests as experience requires.

I would expect nowadays you could fairly trivially produce near life-like footage and allow self driving cars to 'play a video game' that tests tens of thousands of situations as a pre-requisite for certification.

The great thing about computer generated test cases is you don't need real footage of every hypothetical awful thing that could happen i.e. a truck losing control and rolling over sideways. These could be a stage 1 test -- a pre-requisite to real testing. Like the hazards test before you are allowed to sit the drive test with a real drive test employee [1] to make sure you're not a retard.

[1]: https://www.vicroads.vic.gov.au/~/media/images/licences/hpt_...

That may lead to algorithms being optimized for the “video game” instead of real life.

Next step is you put the system on a test track where you throw a wide variety of cases at it.

What happens if it uses lidar (or any other sensor)? I don’t think it’s easy.

Airplane certification uses a combination of simulator exercises and real world flying through prescribed maneuvers.

Mumble mumble Dieselgate.

I think there is likely a common need for a set of standard training models. This may hinder innovation for some time, but it's a cost we should accept when releasing a potentially dangerous technology to the public. It would have the added benefit of multiple companies contributing to a common self-driving standard which could accelerate its development.

That being said, when the first real cars were introduced to the world and later improved upon there were many more fatalities than we're likely to experience with self-driving technology.

As commented in another thread here [0], HAL at Duke University [1] is putting forward proposals for this.

[0]: https://news.ycombinator.com/item?id=16620439

[1]: https://hal.pratt.duke.edu/research

I'm sorry, but this is silly.

Every technology is dangerous. Every technology costs lives to some extent when spread across billions of people. I'm sure forks take more lives each year than self driving cars.

Weigh this against the potential lives saved. I posit you'd be killing more people with drunk drivers by slowing innovation than you'd be saving via luddism.

The first real cars, dangerous as they were, were far safer than horses.

Just sensor data to replay the scenario as fully as possible should be sufficient. Whatever it looks like to humans, there's clearly something in there that can be difficult for AI systems and so everyone should be using it as a regression test.

They need to share situations that made systems fail, and ensure that it doesn't happen with their specific system.

Well, for one thing, I think I have misgivings in part because Uber hasn't really demonstrated an attitude that makes me think they'll be very careful to reduce accidents. (Also, the extent to which they've bet the farm on autonomous driving presents a lot of temptation to cut corners)

Circumstances basically force them into betting the farm. Whether they want SDCs or not, if someone scoops them on SDCs, it's an existential threat to their business.

Sure, and also paying the drivers is a big part of the reason they aren't making money yet. Nevertheless, I don't think it changes the fact that there is a big temptation for someone to start cutting corners if it's not moving along fast enough.

Oh yeah, I'm not disagreeing with you. I think it actually makes the temptation much worse. They've built this business with billions in revenue, and they will probably lose it all if this SDC doesn't succeed.

What's more, a lot of people seem to think Waymo's tech is further along. So not only does this project have to succeed, it is also the underdog. So no wonder they're aggressive.

It makes me uncomfortable too. I think it's because it's a real world trolley problem.

We all make similar decisions all of our lives, but nearly always at some remove. (With rare exceptions in fields like medicine and military operations.) But autonomous vehicles are a very visceral and direct implementation. The difference betweeen the trolley problem and in autonomous vehicles is in time delay and the amount of effort and skill required in execution.

Plus, we're taking what is pretty clearly a moral decision and putting it into a system that doesn't do moral decision-making.

That raises an interesting question: Should there be the equivalent of the FAA for self-driving cars? (Perhaps this could be a function of DoT.)

Although the FAA is of course involved, it's generally the NTSB that is the lead investigator in airplane crashes. The NTSB already has jurisdiction to investigate highway accidents.

There's also NHTSA (which is indeed part of the DoT).

It looks to me like we don't need any new agency at all, just a very small expansion of NHTSA's mandate to specifically address the "self-driving" part of self-driving cars.

Anyone know how this is handled between Boeing and Airbus? Can we mandate the same mechanism between Uber and Waymo?

Thinking about different scenarios as unit tests it shouldn’t be hard for them to simulate all sorts of different scenarios and share those tests. Perhaps that would become part of a new standard for safety measures in addition to crash tests with dummies. In fact, I really think this will become the norm in the near future. It might even be out there already in some form.

> We need to avoid treating deaths as progress in the pursuit of better things

Then by all means lets stay at home and avoid putting humans in rockets ever again because if you think space exploration will be done without deaths you are in for a surprise.

Notoriously criminal company killing pedestrians with robo-cars to make money for themselves != Space exploration by volunteer scientists and service members