We're not doing that while driving, though.
If we're talking purely about going off memory, there's no reason why machines couldn't build up a similar catalog (which could be used by every self driving AI once learned). And human ability to judge distances varies significantly between drivers.
But we do move our heads around pretty frequently. Enough to build mental records of what the bounding boxes are going to be for a range of objects.