This is not a question of reducing complexity, as even very simple statistical tools will sometimes give you high conviction (and yet incorrect) predictions because they are simplified models.
The assumption that error conditions are easier to recognize and control than normal operating conditions is insidious.
For example, an AI could misidentify a patch of ice with very high conviction that it is line marking. A human could too, but we are talking about an AI that has to recognize when it's unsafe and pull over (as you wrote).
I'm sure you've seen the case where an NN taught to recognize wolves did so by picking up the presence of snow as the sole feature in a photo. Only because that happened to be the most distinguishing feature of images with a wolf on it! That AI would only recognize a wolf as long as the photo had some snow. It'd also of course misidentify mere snowy landscapes as a wolf. The mistake is silly, but issues like this are very difficult to diagnose once the inputs become more complex.
A completely non-AI self-driving solution could be cautious and responsible, and sometimes the cautious, responsible thing to do is pull over.
So while what you say is true, I don't think it's relevant to whether or not AGI is required for self-driving.