But it doesn't appear to be helping. Here's an example accident where depth data from Lidar would have helped:

"Tesla later said that during the crash, Autopilot’s camera could not distinguish between the white truck and the bright sky."

https://www.nytimes.com/2021/12/06/technology/tesla-autopilo...

The crash you referenced occurred in 2016 when they were using radar on the cars and I don't believe they were yet using raw photon counts nor did the NN have any voxel-based memory as it does now.

> any voxel-based memory

Haha, any WHAT?

Seriously though do you have any more info on that, it sounds intriguing. Where and how do voxels come into play in a 2D NN?

It is pretty cool: https://youtu.be/ODSJsviD_SU?t=4355

They transitioned from 2D to 3D a couple years ago, major transition but it does seem like a critical step. We live in a 3d rather than 2d world.

Humanity doesn’t know how to solve this yet, so it’s hard to say whether it is helping or not.

We already have the solution. LiDAR.

If you could use LiDAR well enough then it would solve the problem. Of course, if you could use vision well enough it would solve the problem too.

The big limits of LiDAR are cost, more than anything. There have been dozens of public driving trials where from a functionality level the answer has been positive (apart from traffic lights, the bastards), but nobody wants to buy a solution with a six figure BOM, before integration.

Lidar also has problems in rain, fog, snow, etc… FLIR would actually be better

If LiDAR was a solution, we would have driverless LiDAR based vehicles. No one has solved driverless though.

> They basically remove the concept of exposure entirely and simply pass the sensor photon counts to the neural net.

That sentence does not make sense. There's no such thing as a count without a corresponding interval that count occurred over. That interval is the exposure.

You can of course do lots of (very) short exposures to avoid sensor saturation. That's "just" a movie at a very high frame rate. And then you can post-process this in lots of exciting ways, align the frames, average them, etc, etc.

Yeah that's fair. A CCD sensor basically converts individual photons to electrical charges. What Tesla has said they've done is thrown away all the traditional image signal processing & post-processing which often includes a lot of exposure-related averaging.

You're right though that we don't typically use real-time neural networks that operate based upon spike rate, so an interval needs to be chosen for photon counting which could be considered a kind of exposure and it is critical that the interval be short enough to avoid saturation.

Lol this doesn't make any sense. The dynamic range of a fully sunlit California highway during noon in the summer (I.e. the brightness reading of the darket vs the brightest spot) is wayyyy higher than any existing sensor. You cannot ignore exposure, you have to choose which part of the scene you want within the brightness range that your camera sensor can capture. You will have areas of the scene that clip, in other words areas of the scene that are pure black or white with no data.

You can do bracketed exposures, but that's literally the opposite of ignoring exposure.

Just keep the duration low so that you never saturate the sensor even in bright sunlight and let the NN do the summations.

At a fundamental level it is somewhat akin to bracketing except all that HDR processing/frame matching is performed within the NN rather than a traditional image processing stack.

The NN is better at this anyway since it must already be performing camera/pose motion tracking to correlate what it's seeing from frame to frame.

how do you count photons continuously? what... this makes no sense, if you pass "the photon count" you just did a exposure... also how does a photo diode count photons?

Does it have electrolytes as well?

Nice tech and single photons and whatnot but it still runs into things that a radar with some really simple code wouldn't. ¯\_(ツ)_/¯

Counting photons won't keep a camera from being "jammed." Unless you are using a physically perfect polarizing filter, such that each pixel on the sensor only receives photons from the exact angular window, traced back through the lenses, you have a camera that can ultimately be "jammed."

The human eye isn't so great on those terms. But humans can raise their hand to block the sun if it's straight at our eyes.

The replies to your comment don't seem to understand you at all. in the video link here

https://youtu.be/ODSJsviD_SU?t=4424

he clearly states 16x dynamic range as a result of direct photon processing.