Aside from whatever Tesla is doing, LIDAR has some fundamental technical deficiencies such that it is unlikely to ever be a viable long term solution. I find the arguments against LIDAR compelling with the context of having worked with this data.

LIDAR is absolutely the technically superior approach compared to photogrammetry + ML but is cost-prohibitive (in Tesla's view)

Personally, I'll never trust their self-driving product for anything more than lane-keeping unless they incorporate LIDAR

LIDAR would certainly solve the problem of crashes, which is indeed a serious issue.

But I don't see how you could drive with it. It couldn't identify lane markings or exits or traffic lights or cones. And if you can do that well, you should be able to avoid hitting things.

Clearly Tesla hasn't solved that and maybe never will. But adding LIDAR wouldn't really solve the problem.

It's not LIDAR alone it's a combo. The problem of course is that Tesla/Elon is being stubborn and unwilling to put the "expensive" lidar claiming that pictures is enough to work on.

Maybe it is, but not today.

Volvo seems to be getting it right hardware-wise with it's forthcoming EX90 (electric XC90). 250 meter range LIDAR (cleaned by the windshield wipers), 5 radars, 8 cameras and 16 ultrasonic sensors.

That's amazing. Mercedes has the first certified level 3. Only a few states, but the rest will come. Tesla is falling behind due to its hubris.

I also think they haven’t figured out how to do the sensor fusion with the compute available.

their real problem is frame to frame consistency, due to lack of a world model.

this is what Elon talked with Lex a year (or two?) ago. (they mentioned it as how to know you are in a school zone, and remember you are in a school zone, etc)

Sounds like they're working on the world model at least in places. With the removal of the parking sensors, they've started remembering obstructions that are now out of the cameras' vision.

can you be more specific what those deficiencies are?

LIDAR has some issues in rain and snow, and can't read road markings or signs (or make out colours). And it's pretty expensive currently. A combination of LIDAR and cameras is likely what'll end up working best, they complement each other nicely.

Just to nitpick/paint a less absolute picture: Having worked at a couple companies using lidar, you can make out stuff like road markings based on relative brightness (especially if retroreflective paint is used), in ideal conditions, but that's of course not everywhere.

I agree a combo seems like the way forward.

The one that few people think about is that use of LIDAR interferes with other LIDAR. LIDAR works mostly because almost no one is using LIDAR. By contrast, passive optical scales infinitely even if it is more difficult and there is an existence proof that it is possible. Automotive companies view LIDAR as a dead-end even though it works well (in some environments) now.

I see LIDAR self driving vehicles all over the place around Phoenix and Tempe. My Tesla’s autopilot can’t reliably stay on the correct side of the road going through intersections.

Humans don't have Lidar. They can still drive just fine. Solving self-driving is OBVIOUSLY not about Lidar.

Cameras are nowhere near the capability of our eyes especially in low light.

Computers are nowhere near the capability of our brains in terms of image processing, object identification, path anticipation etc.

Until these things improve, self driving cars need to compensate for this gap with more data inputs to be reliable.

Camera are better in low light than our eyes Computers are faster than our brain Software (Neural Net) are not at the level of our brain, in particular they do not fine tune constantly to the current conditions and problems. But they are getting there. Just looking at the visualisation in FSD you can see how accurate the system is at recognizing all the cars, their position, theirs speeds, etc. Human only only track a few objects and only when they catch our attention. Furthermore theirs system has 8 cameras, no distraction, sleepiness, etc.

I saw this visualisation recently in a taxi. It was constantly changing its mind about things. One second a scooter would appear as a garbage bin, the second it would disappear and then show up as a scooter again. Pedestrians would only be displayed while they moved. I was surprised how inaccurate it was.

And cameras are really bad at noise levels at low light and don't nearly have the dynamic range levels needed for good night vision.

Just wait for better cameras? Big pixels solve the noice level problems and multiple cameras solve the problem of dynamic range. People also can not measure distance to static things if seing capability is limited to only one eye (riding bicycle is possible with one eye but driving a car is not).

Depend on version of fsd. They used to go through a normalization process that tried to make the video similar in differents conditions. This has been ripped off and now the signal go strait to NN. I heard (and saw video) where the camera detected animal in the dark that were impossible to see to the naked eyes.

It’s possible their system is not that good, but eventually cameras will outperform the human eyes in every way if that’s not already the case.

Our perception is not that good, but our brain filter it out and make us believe that we see the full picture were if fact we see only what catch our attention.

This is a common refrain among people who are not studying Tesla’s progress particularly closely, and/or are unfamiliar with what LIDAR actually does. Long story short, Tesla FSD still has some way to go before it is ready to be a properly autonomous robotaxi — but where it's failing are not areas where adding LIDAR would help. Tesla's vision stack is already sufficiently capable of mapping the three-dimensional environment with sufficient precision.

Tesla FSD needs better planning strategies when faced with unusual obstructions like novel construction zone diversions. LIDAR can't help there.

Tesla FSD desperately needs more road sign reading skills. LIDAR can't read road signs.

Tesla FSD will eventually need to recognise law enforcement officers and respond to hand gestures. LIDAR cannot disambiguate a LEO from any random person, nor can it translate hand gestures into intent.

The long tail of driving is dealing with the extremely obscure and yet blatantly straightforward tasks. LIDAR is useful for none of them.

Lidar's are excellent at verifying that the 3d model map is correct. E.g. a perpendicular semitruck with trailer on the road.

For item recognition the resolution gets low quick though with range.

You're right, that's something LIDAR is definitely ideal for, but again, Tesla's vision stack is doing just fine here, when observing other people's use of FSD beta.

> Lidar's are excellent at verifying that the 3d model map is correct.

IIRC Tesla does use LIDAR in their test vehicles to measure the ground truth of their 3D model maps. They just don't do that with every single car on the road.

Examples:

https://twitter.com/gbrulte/status/1395356317410287617

https://electrek.co/2020/06/29/tesla-spotted-testing-prototy...

Wouldn't LIDAR be useful to help it not occasionally drive into objects?

Waymo talks about using LIDAR on their gesture recognition system: https://blog.wing.comhttps//blog.waymo.com/2022/02/utilizing...

Is this the same vision stack that was pulling up ahead of stop signs in some places because the signs were bigger than it expected?

If it can't make sense of something as simple as that, why would I trust it with hazards?

https://www.thedrive.com/news/teslas-can-be-tricked-into-sto...

Certainly that is the current Tesla PR spin that their vision stack is good enough to not need it, but it is hard to see why LIDAR wouldn't be useful for defense in depth against the situations like people tricking Teslas into painted tunnels like Wile E. Coyote. Especially when we all know the reality is that LIDAR is extremely patent protected and expensive to buy because of that and Tesla dropping LIDAR was pure cost cutting.

(Self-driving seems to me to be somewhere you absolutely want as much defense in depth and redundant sensors as possible.)

I'd like to see someone exposing a Tesla to a intricately produced Wile E. Coyote tunnel. I would predict that it would stop, because the vision stack leans heavily on motion vectors to map out drivable space. And even if it didn't, the tunnel could only be painted to have the correct perspective at one point along the road. Moving closer would result in the model seeing the road get narrower, causing the car to slow down (narrower roads necessitate slower speed) until it gets within a few car lengths — at which point the painted road would appear so narrow as to be literally undriveable.

If Tesla hasn't already tested this in simulation, I'd be astounded.

Also if a false tunnel good enough to fool FSD beta, I'd suspect it'd also be good enough to fool a decent proportion of human drivers too.

Certainly there is some hyperbole there, but not by much: https://screenrant.com/stop-tesla-autopilot-fsd-self-driving...