Maybe we need an open source protocol that has all the standards and edge cases handled.

I think competitiveness is a good thing and will drive innovation. But in this specific field, with regards to safety and security, it needs to be out in the open.

The protocol is likely uninteresting rather it's the models, training data, and training configurations which are most interesting and the least likely to be open sourced. I don't work in the autonomous vehicle space but I would be surprised there's a set of deterministic rules that are easily reviewed instead collections of deep nn models.

Wouldn't you test the output rather than the input? Before an autonomous vehicle / update is allowed to go on public roads, it must pass a "driving test" that includes a bunch of known edge cases.

Sure. And maybe that's what will happen. But now you have pre-defined tricky situations. Everyone builds those very specific situations into their systems but may not really generalize them.

The data is the real rub. There's some open source code in the space and we'll see what else gets opened up over time. But AFAIK, no one is talking about making their data sets public (and, to the degree that data is coming from regular drivers, there would probably be privacy concerns with doing so). And, as you say, with NN techniques, a lot of actions aren't deterministic or understandable.

>> But in this specific field, with regards to safety and security, it needs to be out in the open.

This is one of the horrors of capitalism.

If there's money to be made, you can bet that if someone creates a proprietary protocol, it won't be released and will force other companies to develop their own technology to compete.

Yes, it will drive competition, but I fear that it will spawn a host of different, competing technologies and make the whole industry awash in proprietary technology. In most of those cases, the people who lose are the consumers.

Volvo's three-point seat belt and Jonas Salk's polio vaccine make good (although old) counter-examples.

I'd be interested to hear if there are more recent examples.

John Deere (I work for John Deere but do not speak for them) freely licenses all safety patents except for ones that we think any competent engineer can work around. We figure we make more money from someone who buys our competitors product and because of a safety feature survives to make the decision on what to replace it with when it wears out; than we make from the person who buys something without a safety feature and as a result dies.

I used to work for the company that published rfc 1223, giving away private information. The motivation is a competitor reversed engineered the protocol, and some customers had both. There was now a need to change the protocol but we knew our customers would not accept having to throw out the competitors equipment to take the upgrade. By publishing the protocol we ensured the competitor understood it and thus when the bit that was "always 0 apparently unused" changed to 1, with a change to the meaning of all following bytes they understood why it was always 0 and why the change happened and how to read all the following bytes.

That's what Comma.ai was supposed to do, but there is no money in it apparently.

Comma.ai wasn't originally going to be open-source, Comma.ai intended was going to be a $999 proprietary retro-fit product. NHTSA (National Highway Traffic Safety Administration) asked Hotz to delay the launch on safety grounds.

He then open-sourced the software specifically to avoid regulation and safety standards.

  "NHTSA only regulates physical products that are sold,” Hotz said.
  “They do not regulate open source software, which is a whole lot more like speech.”

[1] https://qz.com/849669/to-dodge-government-regulation-self-dr...

[2] https://www.theverge.com/2016/11/30/13779336/comma-ai-autopi...

I still like the argument that critical firmware should be open sourced by law.

You're required to be given the chance to read contracts before you sign them, I don't see why software should be different.

Why is software like this allowed to be closed?

We don't allow our plumbing, structure wiring, welding, or any other physical construction method to be closed away and hidden from view. Rather we demand these things be inspected, certified, documented and generally adhere to demonstrable codes and rules.

Keeping you source hidden is saying "I have something to hide". Why should someone have something to hide in Pacemaker software, software driving vehicles or safety system on nuclear power plants or anything else where life could be lost?

We also have a concept of how to inspect plumbing. I could put all rules on a checklist and teach you how to check the plumbing in your house. (I'd have to look the rules up, but conceptually I could).

Of course houses are all the same: a few toilets, sinks, tubs and showers. The differences are easy to handle in the rules. You could imagine something different: instead of water we have a special hand cleaner in this sink. My checklists cover the potable water system and the non-potable water drain system, this special cleaning fluid isn't covered and your cannot inspect it. We have a simple exception for those cases: a professional engineer can legally stamp it saying "I have thought of everything, this will work if installed according to these specs." Then the checklist is make sure that everything is up to those specs.

I honestly do not know how to create a checklist for source code. Sure I can create some starts, but https://en.wikipedia.org/wiki/Underhanded_C_Contest needs to be considered: can someone pass your checklist and still win?

I don't think I have any right to see the blueprint of every building I enter. Ensured certification is very different from open source

I think in some municipalities you can walk down to the planning and get most the blueprints you would like.

Where you can't it is partially because the analogy breaks down physical is slightly reduced by openness, its allows attackers of all kinds to plan better. More manpower can be focused to overcome and issue and a map is force multiplier. Security in information systems is strictly increased, because the flaws exist open or close, but when exposed in the open they are closed forever to all attacker regardless of manpower.

How is software anything like a contract? The two seem completely unrelated to me

IIHS and NHTSA will develop lists of oddities like this that they will test, I'm sure.