Human higher-order reasoning is crucial to our ability to use vision to interpret the world. We have a model of the world and we use our vision to adjust it. We don't just draw bounding boxes, we label objects as cars or trees or shadows, and use our understanding of basic physics to predict their possible movement. We further label other data as humans or driving cars or animals, and use our understanding of other agents to predict their plausible movements.

Vision is only used to periodically adjust this model of the world - most of the time, we are living in our heads. This is so true that we normally don't even notice that we entirely lose our vision every time we blink or move our eyes, and we don't notice the extreme blurriness that we have in areas of our 'vision' where we are not currently focusing our eyes.

It's very educational to visit some areas built with non - common sense objects to understand just how limited human vision is if you remove our understanding of the world from it.