[1] https://docs.opencv.org/4.4.0/d2/d85/classcv_1_1StereoSGBM.h... [2] https://www.intelrealsense.com/stereo-depth
People tend to think about machine vision in perfect conditions. This is the problem, me thinks. In practice, you will get a lens flare in most important case, or lose a sensor or get some rain.
Instead of designing solutions starting from most hardcore edgecases (which are common with humans), robotics researchers tend to provide MVP that works in best situation. That is far from end-user environmnent.
This needs global fixes, if you (I don't care) want robots working in real world.
That's a "we need an expert in filter theory" problem. I had four people crash and burn on that problem when we were building a DARPA Grand Challenge vehicle. Combining GPS, accelerometer, gyro, compass, and odometer data to get position is a hard problem. All those sensors are noisy, but in quite different ways. There are off the shelf solutions now, but there were not in 2004. We could not get below 3 degrees of heading noise, and had trouble keeping the sensor map aligned to the real world.
I'm certainly not claiming that I had a fully robust solution but I'm pretty sure lens flares wouldn't be too hard to filter out and there are lots of useful things that need doing in rain-free environments.
I worked on this a bit a number of years ago and I thought I had scene matching working pretty well. The problem is I was trying to make it work without actually having two cameras (ie. on a smartphone where binocular cameras were not available at the time and for the most part still aren't). I was hoping to use the accelerometers and other sensors for dead reckoning the camera's position as the user swept it over a scene, but that turned out to be too hard.