The real way is to use the LIDAR to add to the depth probability distribution in the stereo depth estimation. This way you aren't throwing any data away. LIDAR often gives probabilities as well, for example, and this can be used to eliminate reflections.
Reading sensor data is not the same as feeding that data to a neural network and asking it to form a worldview composed of possibly conflicting sensor data streams(i.e. lidar vs vision vs ultrasonic).
You are somewhat correct that it is quite trivial to read sensor data. For many sensors, there is some work which needs to be done to denoise or cleanup the input data. That's not where the story ends, however.
Sensor fusion is computationally cheap. It's just a lot of R&D to do it in a way that leads to a net gain in precision and robustness.
0. https://sensor-js.xyz/demo.html