I also was thinking, perhaps all memories aren't visual or auditory, and I then I read your post. It's quite possible we all have memories of that time in our lives, but they are emotional in composition rather than comprised of physical senses.
Is the ability to recall something specifically a 'memory,' or could emotional feelings with seemingly no connection to the present also be considered memory? Like a lifetime of emotional deja vu.
When you recall something stressful, and you feel stress in the present, is that stress you are experiencing from the present as your brain processes a stressful scenario in your mind, or is that stress part of the memory, and your brain is re-experiencing that stress in the same way you can visualize a memory?
Later in life these same sensations evolved into pretty much nightly existential dread, and it's taken me decades of "I'm not thinking about this" to not have panic attacks about inexistence and the extreme temporariness of my self.
Due to my early life experiences I never really dismiss a baby crying as something inconsequential like "awww he's [hungry/tired/whatever]!" because for me I was often experiencing some really primal variation of existential terror before I could even speak.