> If you want to do it wrong, you start out with lane keeping...recognition of common objects...some machine learning....hope for the best

Thank you! Can I ask why the self-driving car programs like the 3-semester Udacity one deliberately focus on going about it the wrong way ? Why spend 1 whole semester on lane detection using trivia like OpenCV, 1 whole semester on object classification using CNNs with Keras, spending so much time on combining lidar & radar data using Kalman filters for moving object tracking .... surely all of these are peripheral issues and the wrong way to go about the business. Then why ? Is it all one big clueless scam ?

To be fair, I am enjoying the course very much as a student. But it's becoming clear from talking to colleagues in the autonomous car industry that what is being taught is not the real deal. This isn't what happens in production, so to speak.

As a Udactiy SDC student finishing up the program without much real insight into the industry, the program seems very much to me like an entry-level overview of topics you're likely to encounter in this field: computer vision, machine/deep learning, path-finding, sensors, and some basic system integration.

By the end, you'll have a few toy projects and a very basic ability to converse about or study more deeply these topics. Maybe they should have spent a little more time on maps and SLAM instead of other stuff, but hey, it's their first version anyway; I'm sure they'll iterate (as they already have somewhat in response to feedback). They did kind of admit in the CV introduction (lane finding) that this was kind of a super simple, satisfying project to start with. I feel like they often touch on topics acknowledging they have more of a historical relevance and are no longer state-of-the-art tools.

(Edit: I should also note: while I have criticisms of the program, overall I'm very happy that I took it. I'm sure some people are better at seeking out specifically what information is relevant to dig deeper into, but I really appreciate the broad overview kind of pointing me in the right direction. Since starting the program I've amassed quite a list of resources to dig deeper into, and hope to eventually find work in this or related field)

All those subjects are useful. Some version of all those things will appear in the good systems. But it's the difference between building the flight control system for a toy drone and the flight control system for a Boeing 797. You need the "failure is not an option" attitude of aerospace. You need fault tree analysis. What if sensors disagree? What if a sensor is misaligned because someone bumped the vehicle while it was parked? You need enough of an organization to have people working through all those situations. You don't get to blame the user for your mistakes. A "minimum viable product" will kill.

Self-driving is something many people in software haven't seen much - software that isn't feature-oriented. As someone from Google pointed out, the output is just two numbers.

Isn't the practical reality that any course like that will address the theory and essentially deal with more or less toy problems? It would be hard to get into cutting edge research in an online course.

I'm not sure learning about natural language processing or image classification would be all that different--although those problems are arguably better understood and more bounded.

Even a site reliability engineering course isn't really going to give you deep insights into what, say, Google does on a day-to-day basis.

>Even a site reliability engineering course isn't really going to give you deep insights into what, say, Google does on a day-to-day basis.

I think this really gets to the core of the issue. There aren't many courses that will accurately reflect the actual state of the art in any field. Sometimes, graduate courses with a small scope and a relevant professor taught in person can partially address the state of the art, but an online course about "how to self driving car" is not that.

Lectures, tech talks, blogs, random experts in relevant subreddits, but yes, an online course is not going to give more than the fundamentals and some advanced, but already well understood examples.

Chris Urmson's SXSW video: https://youtu.be/Uj-rK8V-rik

Timestamp for duck part https://youtu.be/Uj-rK8V-rik?t=26m11s

Thank you! It's so good to see hype free discussion of this stuff.

I would also add that using simulations to test (or train) robots is also not new, and has significant limitations. Robotics has been using simulation since before the Grand Challenge, but what we find is that simulations make simplifying assumptions that don't bear out in real life: simulated sensors are more idealized than in real life, object recognition is idealized, actuators are idealized. But that makes it hard to apply any learnings to the real world, where sensors hallucinate and actuators lie all the time.

And yeah, we _could_ be fuzzing the simulation and adding failure modes, but not many people do. It's hard to simulate these kinds of defects in a usefully realistic way.

I work in robotics, and I meet people with this attitude all the time. This kind of thinking is OK when your robot is a small desktop toy, or a humanoid robot moving at 1 cm/second, but this approach will not work if your robots can damage expensive equipment or kill people.

The key thing to understand is that your simulator is not a fixed product provided by a third-party. It grows organically with your codebase. So any time you encounter a bug in the real word, you have a three step process:

  - Create a test case to reproduce your bug in the simulation. This may involve updating your simulator to simulate the new physical properties we care about (such as new failure modes or new types of the sensor noise)
  - Fix your code, and make sure your test case passes
  - Add the test case to automatic testing suite and keep it there *forever*. this is the most important part.

For an SDC company, the test suite is almost as important as the code itself. The time spent on writing testing code and test cases could be actually more than time spent writing the driving code itself.

Unfortunately, many academic courses use an opposite approach: they use pre-made simulators, and do not modify them at all. This gives students the wrong expectation that you can find a simulator which can just simulate everything perfectly out-of-the-box; when this (predictably) fails, they become disappointed in the whole simulation idea.

One of the things that seems to be going on is that, after relatively limited visible progress for a long time, the ability to collect and analyze large datasets suddenly produced fairly striking results. And this in turn has led to a lot of thinking along the lines of "We just need to collect more data and crunch the numbers a bit better."

As you suggest, for at least some classes of problems, we'll find that this approach will asymptotically approach some level of "good" that turns out to not actually be good enough. And the ways to get beyond that will be hard and may not be obvious.

That's what seems to be happening in the various image classification competitions. Accuracy has reached 95-96%, but further improvement is tough.

Is there a model ponderated by momentum ? Decisions have different consequences if you're at 30 vs 50mph.

I'm taking notes, go on.