I'm not an expert myself, but I think you overestimate how much ML is used in these systems. From my understanding, it's mostly used in classifying objects, but the actual controls are not, and I would assume that the driving logic itself is written in a conservative way and a bad classification wouldn't cause an insane behavior.

I have talked to a number of different autonomous driving startups and every one of them was entirely basing their approach on NNs and explicitly avoiding explicitly-coded algorithmic approaches. It was crazy and frightening that these people are serious.

Yes, it's scary.

We had an entry in the DARPA Grand Challenge in 2005. It was too slow, but it didn't crash. We profiled the road ahead with a LIDAR, so we had an elevation map of what was ahead. This is essential for off-road driving, and it's the gold standard for on-road driving.

But there's a range limit. Not from the LIDAR itself. From the geometry. If the sensor is 2m from the ground, and you're looking 30m out, the angle is so shallow you can barely sense elevation. You can't see potholes. Anything that looks like a bump hides a lot of road behind it.

The Stanford team's answer to this was to use machine learning to compare the near road texture, which they could profile, with the far road texture, which they couldn't. If the near road was flat, and the far road looked like the near road, they could out-drive their profiling range. If it didn't match, slowing down brought the stopping distance down to profiling range, and they could work through difficult terrain. The 2005 Grand Challenge didn't have much difficult off-road terrain. No rock-crawling or cratered roads. The 2004 course was harder, and nobody got past mile 7. So most of the time, fast mode was usable.

Google/Waymo started from the Stanford approach, trying hard to profile the immediate surroundings. Their machine learning efforts were focused on identifying other road users - moving objects, not fixed ones. Their earlier videos make this clear.

Google/Waymo built a cute little bubble car with a top speed of about 25MPH and a LIDAR on top. At that speed, you can profile the terrain all the way out to your stopping distance, so you have solid detection of fixed obstacles. That was something that had a good chance of working well. They decided not to manufacture it, probably because it would cost far too much.

Machine learning isn't that accurate. You can get to 90% on many problems. Getting to 99% is very tough, and getting to 99.9% is usually out of reach. The killer apps for machine learning are in low-accuracy businesses like ad targeting. Or in areas where humans have accuracy in the 80%-90% range.

Here lies the problem. Humans are very good at obstacle avoidance. Much better than 99%. Machine learning isn't as good as this problem needs.

There's another side to the problem - the false alarm rate. If you build a system which insists on a valid ground profile, a piece of crumpled cardboard on the road will result in slowing down until the sensors can look down on the cardboard and see past it to solid pavement. You get a jerky ride from a conservative system. That's why Uber disabled automatic braking and killed a pedestrian. That's why Tesla's system fails to react to fixed obstacles it could potentially detect. Waymo has struggled with this. Customer evaluation of driving quality seems to be based on good lane-keeping and low jerk. These are things that trouble poor human drivers. Self-driving has different strengths and weaknesses. This is what leads to systems which seem to be doing great, right up until they crash.

What self-driving seems to need right now is a rock-solid way of detecting the lack of obstacles ahead. All we have so far is LIDAR. Radar is still too coarse and has trouble distinguishing ground return from obstacles. Even LIDAR is rather coarse-grained. Stereo vision doesn't seem to be hugely successful at this. We need that before self-driving vehicles can be trusted not to run into obvious obstacles.

If you have to recognize what the obstacle is before determining that it's an obstacle, it's not going to work.

There are a whole series of secondary problems, from left turns to double-parked cars. But those are not the ones that kill people. It's the basic "don't hit stuff" problem that is not adequately solved.

Do you have any references to substantiate anything you have said?

Chris Urmson's talk at SXSW is good for how Google/Waymo's system worked. The DARPA Grand Challenge is well documented. Our team's code is now on Github, just for historical interest.[1]

[1] https://github.com/John-Nagle/Overbot

In your opinion, which company is taking the best approach to solving this problem?

Don't know. Nobody in commercial self driving publishes much in the way of technical details any more.

(That's part of the price of hostility to software patents - more trade secrets.)

Having actually worked in these companies, I can say the actuation and controls are still deterministic algorithms. Talking on some buzzword is one thing, but actual implementation is another. I would be surprise if any of these companies have largely NN based modules in their stack for controls and lower level planning.

Can NNs be corrected, like a child or a dog being told a firm "no" when doing the wrong thing? Can the conditions be replayed such that those conditions result in a different response with a human auditor providing the correct response?

I'm guessing the answer is currently no. Which is interesting because one of the early benefits touted for self-driving cars was that people might die, but a patch would ensure no one dies twice for the same reason (within reason, which is more than can be said for humans).

The even scarier thing is that a lot of these startups are training their neural networks on game videos like GTA or cars they get off the Unity asset store. I seriously doubt any of the artists, graphics programmers, AI developers, etc. involved in these titles think their work is suitable for safety critical systems.

the positions for Vision/Perception at zoox all mention nn's: https://jobs.lever.co/zoox i bet even simple questions like "how does this scenario change decisions when the lighting / sensor rotation are altered" cannot be answered.

Changing lighting and rotation on sensor input is a pretty standard way to improve neural net performance (it's called Data Augmentation), so I'm pretty sure they could answer that.

The Telsa autonomy investor day (which applied to the cars not the people) spent most of their time explaining their NN chip, ghost riding to correct predictions etc. Definitely gave the impression that the network was doing a lot of the decision making.

Of course, because NNs are hot with investors now and Tesla wants their money.

Companies frequently misrepresent how their technology works to the public. Typically they'll do some small portion with the hot technology for buzzword compliance, and then build the rest of it with an actually sensible, boring, and tailored-to-the-problem technology stack, oftentimes with a lot of proprietary legwork done by their data scientists and engineers. This way they get the best of all worlds: investment dollars from gullible investors, PR from journalists that want to hop on the next big thing, a product that actually works, and misdirection so competitors hop on approaches that aren't going to work anyway.

That doesn’t seem to be the case with Tesla. Their custom chip is almost entirely focused on being a neural network accelerator and they seem to be hugely reliant on neural networks. Notably Waymo is said to be less reliant on neural networks but Tesla definitely depends on them.

That doesn't really prove much. The classification is the heavy part that needs acceleration. The algorithmic side would probably not need an accelerator, as it's basically a decision tree.

My original point was that the decision tree should hopefully degrade gracefully as the prediction quality goes down, and not have any brand that leads to an insane behavior.

Now obviously, the better your predictions, the better the driving will be, which is why you'd want NN accelerators.

Gives me the impression that Tesla made a small innovation which they're trying to overblow to investors since buzzwords like machine learning are more interesting than actual progress. I'm not saying Tesla isn't making massive progress but that anything they say to the press should be taken with a grain of salt.

Why speculate? Watch the I vestor day video and see for yourself why it's not just hot air.

> I would assume [...] a bad classification wouldn't cause an insane behavior

I want to assume this too, but what evidence is there that we can actually assume this?

Well if the system was as I described it, you can throw as many bad classifications as you want in the simulation environment, of varying severity, and see how the algorithmic side reacts.

Tell that to the owners of Tesla vehicles that have been running into stationary fire trucks.

Where are you even reading this assumption? Who, which company has said this? Sounds like something someone overheard somewhere, and not an actual assumption made by any of those companies. In reality Waymo, Tesla and others are using NNs where they fit, where they have been shown to produce the best results - and other methods everywhere else. Heck, even the ML courses teaching self-driving tech to students don't exclusively use NNs, but all sorts of ML models.