What you describe echoes with my experience.

I've had conversations with family members that confirmed I was somewhat unique in experiencing this way too. Most didn't relate, though one brother is waaaay more synesthetic than I am.