It's kind of (but not exactly) like proprioception. Close your eyes and hold your hand out. Even without visualing (because we can't...), you know "where" it is. It feels a lot like that. I just have a good sense of where things are relative to me and each other.
I am bad at visualizing things from what I can subjectively tell, but as far as memory is concerned it's mainly the spatial memory tests where I perform poorly. So it's hard for me to imagine what recalling spatial memories is like, besides the assumption that people can probably visualize them.