IMO we do - try driving with the windows on both sides down a crack. Your brain synthesizes traffic noise into your mental model. You can be aware of traffic near you without being able to see it.
You know how your eyes can sometimes get things wrong? For example, you can see a mirage of an oasis when you are in the desert? Well, radar has failures like that. For example, in normal everyday driving conditions, like when you go over a pothole, someone passes through the lane in front of you, or you go under a bridge radar tends to hallucinate just like your eyes tend to hallucinate. It tells you things that aren't true about the observed reality.
The most famous story about radar that I know is the time that ignoring it prevented the death of humanity. There was an illusion of a nuclear launch by the United States on Russia. The officer didn't believe the radar and so didn't launch nuclear weapons in retaliation.
If you were combining radar with your eyes while driving, rather than combining it with hearing, you would have times like that - times where you recognized that the sensor was wrong and you were forced to ignore it. Or you would die. And others might very well die with you.
A computer doesn't "forget" to look before merging. A computer doesn't drive drunk, fall asleep, get road rage, or get distracted by a phone. A computer can see in all directions at once with enough sensors.
I 100% believe that a vision-only self-driving system CAN work...but that radar/lidar providing extra signal would make it a lot easier to implement, especially at night when oncoming headlights can be blinding.
If you use the average rate of error and you combine it with the vision reading the sensor fusion is much worse, not much better, and in practice it resulted historically in break checks at underpasses, followed by drivers taking over to prevent themselves from getting rear ended.
In theory you could absolutely do sensor fusion, but it isn't trivial. You need to have a network that can (1) understand the scene and (2) understand the error distributions as they relate to each particular scene. But notice this - vision is usually more reliable than radar, by a hell of a lot, so how exactly are you going to be determining the edge cases where you need to assume a different error distribution for radar? Seeing how tricky this gets? Combining with radar relies on vision being reliable enough to help inform you of how radar is going to fail. Ugly circular dependence right there.
Lets say you can compensate for the potential for radar to error dramatically. In theory, this would allow you to have a better reading in some edge cases, but in practice those edge cases don't matter. What are the actual edge cases where it helps?
1. Driving in the dark or while blinded: radar isn't sufficient to drive safely. It can't see lane markings for example. Therefore this isn't a solution. The correct thing to do if you can't drive on vision alone is to not drive. Or - and I'm dead serious here (fully automated logistics is basically so ridiculous for enabling massive wealth that very unrealistic things are worth pursuing to attain it) - modify every road in the world so as to trivialize driving with sensors other than vision.
2. Seeing obstacles that are hidden from sight. This is an edge case that matters and is where you would get the big win, but it isn't actually just a win. It is also a huge complication. Uncertainty matters and you just made it very hard to project a cone of occlusion producing uncertainty because now we are claiming knowledge of occluded areas with an error prone sensor. You need some feedback mechanisms here or you are going to underestimate/overestimate your ignorance in situations involving uncertainty, potentially resulting in bad driving decisions as a consequence.
3. The more complicated you make this, the greater the potential latency to make the decision. Pursue the theoretical best and latency could increase enough so that you delay your decision. However, this is a real time system. It can't afford pure theoretical best, because latency matters a lot. That makes the sensor fusion more useful as something you use in the backend systems that aren't real time - training time and inference time have very different properties.
I actually basically agree with your larger point that in theory these other sensors should be able to benefit. However, I disagree that getting them to the point where they do benefit is easy. I think it is actually pretty hard.
—- 50% of humans after a crash