I didn't watch the video, only the slides. But got the impression with this approach it would be pretty much impossible to create safe and reliable self-driving technology. At some point chasing a long tail of unusual events will simply become too costly to continue, but the technology still won't be safe enough to use in the wild.

Self-driving will always be held back by municipalities doing weird things with their traffic lanes and lines (and cones and hazards and signal trucks and road construction) as long as the AI puts as much faith in the lines as it does. And you'll never track down everything because construction is constant in the world, with the possibility of introducing a novel road-weirdness.

It's a road system designed for humans, and with current tech, you'll always have issues chasing the long tail down. Though I'm fairly sure self-driving is still statistically better than humans behind the wheel, especially distracted humans as the information age has made us.

The more immediate solution is white-listing safe roads that have sane paint lines and are relatively straight-forward roads. That way, the chance of the AI getting it wrong is drastically reduced. This covers most highways where you get the most out of self-driving systems anyway. Bonus points if you can automate this decision-making with recent satellite imagery.

Maybe the responsibility (and liability) for accidents due to weirdly designed or badly marked lanes would have to fall upon the municipality. You could make an analogy to trains: if a local government commissioned a rail line that included a junction that can't be navigated without derailing 1% of the time, you would not blame the trains for not being able to deal with it.

Another analogy could me made to data entry: you could allow free-form text input and try to process all of the long tail of unexpected inputs, or else set out a format that the data has to follow and enforce validation at the input.

> as long as the AI puts as much faith in the lines as it does.

The need for lines is only a temporary crux. AI systems will continually improve until the white lines are guidelines only. There are lots of other factors in an image that can be used to deduce where the lines should be.

But there will have to be handling of uncertainty which humans have to do too, simply just slowing down (not sharp braking) will help most cases.

I share your worries, as I too think that the abstractions used in the trained models in the talk are inherently leaky, meaning that they do not represent reality to a suffiently high degree (see corn flakes vs ketchup screen wiping action). As we do not (yet?) understand the underlying nature of those "2.0-ish" decision algorithms, we will not be able to make a judgement call when they will work, and when they will fail.

The main point you offer is that models can't be statically analysed and will do things unexpected that result in loss of human life. Here's my response:

- Humans also do unexpected things, like stepping on the gas instead of the brake. If Autopilot does unexpected things at 0.01% the rate that humans do unexpected things, then it is a huge safety bonus to use autopilot.

- How can we solve image recognition any other way? We must move the needle forward on our technology. If we do not struggle against the adverse side-effects of our software and make it better, we will never advance it and we will be stuck requiring human drivers for all driving tasks.

> If Autopilot does unexpected things at 0.01% the rate that humans do unexpected things, then it is a huge safety bonus to use autopilot.

You’re forgetting a variable - the frequency or likelihood of a situation to occur. Go far enough down the long tail and neural AI can get far more deadly than human drivers.

It's unclear what your argument is

I’ll try to clarify. In my opinion, the weakness of neural systems is their inability to deal with input for which they have comparatively little or no training. There’s no way around that, except by introducing structures or systems outside the neural nets themselves that provide logical frameworks for dealing with rare events. (And I don’t just mean a bunch of logic in code. Expert systems are one potential approach, for example.)

Your point was that mosern autonomous driving systems can drastically reduce fatalities over human drivers—-and I agree, but only for circumstances for which the car’s neural systems have been well-trained. But the systems ought to be able to handle ~99.99% of the types of circumstances gracefully before most of us will trust them to safely drive us around.

>Humans also do unexpected things, like stepping on the gas instead of the brake. If Autopilot does unexpected things at 0.01

But the difference is, one human doing the wrong thing does not mean every human Will do the same thing, given the same scenario. But whereas a software running on all cars will exactly do the same wrong thing given the same input. Sure, three is also an advantage (arguably), as fixing it once fixes all cars but that has other problems in taking on the update.

You've got a point, the only problem is that we know when humans are more likely to do errors : stress , night time, long hours, high traffic, etc.

With AI we could enter a world where you'll have 0.0001% of having an accident, but this could happen anywhere anytime in any random situation (such as a specially shaped cloud in the sky).

This is what makes it unacceptable, IMHO.