I don't disagree, both humans and ML are bad at classifying things they haven't seen before[1]. However that reasoning doesn't disqualify either vision only auto driving systems or machine learning.

Both statements are true:

"Computer driven cars may crash, even fatally, when they encounter a situation that they do not recognize." and

"People driving cars may crash, even fatally, when they encounter a situation that they do not recognize."

The success criteria for self driving cars is that they can drive at least as well, in the common case, as the set of human drivers who are defined to be "good" drivers. And self driving is not invalidated by a computer's mishandling of an event that a good driver would also mishandle.

I expect that self driving systems will be differentiated by how well they handle the unusual cases so a Mercedes system might do better in an unusual situation than a Chevy system. And all of this discussion is orthogonal to LIDAR :-).

[1] http://puzzlephotos.blogspot.com/

There is a difference though - humans understand the surrounding state, computer vision is not quite there. It can recognize things, and in NVIDIA's case directly generates steering commands without going through the intermediate step of building a model.

Humans build models of the world, and such models allow us to predict the future to a little extent, and explain the reasons behind a situation. Humans can intuit the intentions of other drivers and the behavior of other objects. AI can't do that quite as well.

Also, making eye contact and being waved through. Humans are excellent at reading cues such as this.

You've hit on a key insight. Predicting the future (even by a bit) turns out to be a very powerful learning signal for building models of the world.

It won't work on a traditional feedforward neural network but if you have feedback everywhere it appears to work.

> The success criteria for self driving cars is that they can drive at least as well, in the common case, as the set of human drivers who are defined to be "good" drivers. And self driving is not invalidated by a computer's mishandling of an event that a good driver would also mishandle.

This comes with one important caveat: these are the engineering criteria. The criteria of public perception, unfortunately, may not allow for a computer driver that makes the same mistakes that a human "would have", because people tend to mis-estimate what they or another human "would have" done.

But the computer vision systems can be endlessly improved and merge experience from millions of cars, while human drivers accumulate experience from a single driver, age, and are eventually replaced by younger, inexperienced drivers.

Soon enough these systems will have data from encounters with far more varied situations than any single human will ever be physically able to encounter in a lifetime.

Not sure I see how that would work when there is no 3G signal. If a computer on-board a vehicle sees something it does not recognize when it's not connected to the Tesla HQ, what should it do? And even if it is connected, uploading video over 3G is too slow for the real-time classification needs. Right?

The notion isn't that they phone home to immediately ask about unknown data, but that they periodically feed back any unknown data (or, in the worst case, it gets extracted from a blackbox if a car has an accident), and receive revised models to process the world with.

No, this won't necessarily save you if you see some impossible scenario like a boat cruising down the highway toward you, but it does mean that the list of conditions the model can't respond to reasonably will rapidly trend toward "fewer than most human drivers", in the ideal case.

This will, invariably, have some unfortunate bumps when people discover real-world conditions that, for whatever reason, the model doesn't remotely have responses for (I wonder if they've trained it on e.g. an enormous wall of water, or tornadoes?), but that's why you don't claim it's an always-on self-driving system (e.g. you have to be ready to take over at any point), and arguably the error rate is still going to be lower than most humans to start with.

You need humans to label those "millions of experiences." The bottleneck is not raw video. You need humans to label that data. Otherwise it's useless.

No, you don't. You need to process enough of it to see how the majority of human drivers act in situations where the automated system currently would react substantially differently.

You've just described end-to-end-learning (e.g. the data is raw video/camera/radar/sensor data, the labels are steering angles, etc.)

No autonomous vehicle manufacturer uses end-to-end learning. The only one to claim to use it was Comma.ai, and we all know how that went.

All the autonomous car companies will manually label the camera images - e.g. given an image, draw boxes around where all the cars are.

Learning how to label camera images and how to respond to already detected features are separate issues. You can make the choice of whether to apply unsupervised training to either one separately.

This [1] blog post from Tesla explicitly claim that they are using unsupervised learning as one of their strategies to determine appropriate system response to specific detected objects in specific locations:

> This is where fleet learning comes in handy. Initially, the vehicle fleet will take no action except to note the position of road signs, bridges and other stationary objects, mapping the world according to radar. The car computer will then silently compare when it would have braked to the driver action and upload that to the Tesla database. If several cars drive safely past a given radar object, whether Autopilot is turned on or off, then that object is added to the geocoded whitelist.

> When the data shows that false braking events would be rare, the car will begin mild braking using radar, even if the camera doesn't notice the object ahead. As the system confidence level rises, the braking force will gradually increase to full strength when it is approximately 99.99% certain of a collision. This may not always prevent a collision entirely, but the impact speed will be dramatically reduced to the point where there are unlikely to be serious injuries to the vehicle occupants.

[1] https://www.tesla.com/blog/upgrading-autopilot-seeing-world-...

Classification is not the right metric to use here. Lidar doesn't classify the objects it's looking it, it just tells you the direction and distance.

Cameras can also gauge distance pretty effectively from parallax. Either using multiple cameras, or from the motion of the vehicle itself, or both. From this it should be possible to gauge where obstacles are and drive safely.

But NNs give the possibility of gathering much more information from recognizing objects. Information that Lidar systems don't have.

> But NNs give the possibility of gathering much more information from recognizing objects. Information that Lidar systems don't have.

Wouldn't you feed both the depth map from the lidar and imagery from the cameras into the neural network? I imagine that a variety of different sensors as input would make it easier to do classification. As an analogy, someone who has lost their sense of smell might have a harder time telling the difference between a clean sock and a dirty sock than I would.

Please let me know if I'm wrong here, but I assume that the depth information that can be derived from parallax is not a superset of what you get from lidar (I'm thinking about low light, glare, objects with complicated geometries, similar-colored objects obscuring each other, etc).

You could do that. The advantage of using purely cameras is they are cheaper and simpler, and don't stop functioning during bad weather.

You can use a model that gives you its classification uncertainty. Bayesian SegNet for example [1]. We may also adapt the legislation for how vehicles should look like we did to make human driving easier (ex: tail and side lights).

1: https://arxiv.org/abs/1511.02680

> Humans may not always see a white truck in a snowstorm, but is computer vision going to see it either?

So you put in your training and test dataset a bunch of such situations. At some point you've covered enough cases to extrapolate the rest.

Good testing is going to hunt for these blind spots and fix them. Fact is that it's already safer than humans, even with all its hidden imperfections.

What if that point is 20 years from now? What if every time Ford/GM/Toyota substantially changes the look of their cars, your classifier no longer recognizes them because all your data only has the old models in it. That's what people are driving at. Simply collecting more data is not enough to solve this problem.

At a certain point it's just about recognising an object which shares broad characteristics with a car rather than aesthetics. Eg it moves at the speed a car moves at, it's in the road, it's overtaking on the right hand lane. I would expect any autonomous car to be able to fail over to "this object is likely a vehicle I haven't seen before" given a strange car-like object being detected.

Great. Now the problem you've posed is no longer image classification. It's more like video classification or zero-shot classification! (neither of which are close to solved)

It doesn't seem like zero-shot classification to me. It still seems like image classification. You said:

> What if every time Ford/GM/Toyota substantially changes the look of their cars, your classifier no longer recognizes them

My answer was probably incomplete, but I took the above to mean that cosmetic changes to vehicles mean that classifiers no longer identify them as cars, and this detrimentally modifies the behaviour of the car.

Whilst it's trivial to envisage a scenario where your problem is solved systemically (sufficient training data for a new chassis released in advance or something), it seems like it would be possible to train based on "things we expect to see from any car".

As far as I know, that's how all of the existing methods operate. They seem to have a hierarchy for decision-making:

0. Is there an object around me which I need to consider? If not, continue to monitor for one whilst operating the vehicle within the parameters of road signs and conditions.

1. Is this an object which has a predictable path based on either its signals or the expected behaviour of a car on this part of the road / operating within the parameters set by the road signs I can see?

2. Is this an object which is operating safely despite not falling into category 1?

3. Is this an object which I need to take action to avoid?

Which is to say that it ought to be possible to "fool" a Tesla with a non-car object behaving in a similar fashion to a car. The Tesla sees an object, not a car.

"it moves at the speed a car moves at, it's in the road, it's overtaking on the right hand lane" is video classification, which is not solved. In fact, at least how you described it (you could probably change the problem statement to avoid this), this would involve an ML model that must learn a model of physics - also unsolved.

You've just specified a manually hardcoded set of decision rules. This is not machine learning, and is incredibly brittle.

I think we're talking across one another.

I had thought that in your original post you were agnostic about the methodology for identifying a car, but were remarking that, in a world where it's possible to do it using whatever form of classification, it would be possible to 'stump' any reliable model by modifying the appearance of a car. I'm observing that any model for classification almost certainly would not rely on aesthetics.

> You've just specified a manually hardcoded set of decision rules. This is not machine learning, and is incredibly brittle.

I'm pointing this out to illustrate that the technology already deployed to solve this problem does not get confused by aesthetics.

I was talking about deep learning. The comment I was replying to was making the specific problem seem as if it were easy. Certainly there may one day be a classification technique that does what you say will do. But you may as well have said there will one day be a perfect classification technique that will just perfectly output steering angles, end thread. What use is there in conjecturing about perfect unknown classification techniques? Not to mention that there is no guarantee such a perfect method would not rely on aesthetics. Even if the train set has more than just aesthetics (e.g. video of cars in motion) maybe this perfect classifier would just cheat and rely on aesthetics, you don't know.

So I'm pointing out the methodology you suggested is not currently feasible, or is currently widely considered by the community to be the wrong practical approach. Because theoretical solutions will not solve self driving cars.