'visualizing' isn't just 'seeing'. Try to think back in your mind to the house in you grew up in. Do you remember where the kitchen, bedrooms, and/or bathrooms were? Do you remember which way the beds/furniture were facing in most of the rooms, which side the sinks and counters were in the bathrooms and kitchens? Do you remember where in the house any other furniture, such as a desk, couch, television, or coat rack was? Most people can remember these things, even if they can't conjure forth before their mind's eye a vivid mental picture of their house.

Try answering all of those questions about, say, any of your neighbors houses growing up which you may have been in once or twice. About a recent home, building or room in which you may have only been in once, sometime within the past 6 months to a year.

While it's not perfect, and some particular facts might elude you, most people will find it surprisingly easy to answer most of these questions, even about buildings they may have only been in once or twice a decade or more prior. Yet, if they were to try to, say, answer detailed questions about a painting they may have seen around the same time, most will struggle.

We seem to have a certain kind architectural/location memory which is used for remembering the relative layouts of places we've been, and this sort of memory seems to have some different properties compared to just visual imagery. It seems to be retained long-term fairly effortlessly, with very little time actually spent 'memorizing' it.

This is the basis of a lot of the tricks used by memory champions: https://en.wikipedia.org/wiki/Method_of_loci