So I do a lot of work with both vision and lidar data, and am pretty intimately familiar with the public state of the art in terms of perception systems that rely on each or both, if not planning.

I don't think you need lidar.

I'm not saying that vision-only won't take longer, because it will, but I believe it'll probably happen unless the cost of lidar falls a great deal.

Have a look at, for example, "learning to see in the dark" (the CVPR 2018 one), DensePose (the FB research one) to get a sense of how quickly difficult problems are being solved.