Yeah, me too; I was in the 2005 Grand Challenge. Vision processing has made enormous progress since then. Here's Mobileye's guy explaining what they do and what needs to be done.[1] Here's the NVidia guy.[2] Here's Chris Urmson from Google.[3] All three use their vision system to draw boxes around things that look like obstacles. Then the planner uses those boxes to construct a path. The vision system just gives you a rectangular box in image space; the LIDAR based systems turn point clouds into simple 3D solids.

These vision systems do OK on things that look like typical road obstacles - cars and people. Other stuff, not so much. Only Urmson from Google talks about that much. The videos from the vision systems don't seem to be doing much with road edge clutter. They're not marking every post, fireplug, and low-hanging branch. Train a deep learning system on cars, trucks, pedestrians, and bicycles, and your system sees only cars, trucks, pedestrians, and bicycles.

These vision systems do a lot less than many people think they do. Watch the videos.

[1] https://www.youtube.com/watch?v=GZa9SlMHhQc [2] https://www.youtube.com/watch?v=2NGnvGS0AtQ [3] https://www.youtube.com/watch?v=Uj-rK8V-rik