It doesn't matter why you're closing yourself off.
You start wondering about the why, you'll set out on the path of trying to figure out what people are thinking ... and let me skip to to the end, that path is a dead end. People feel and think things that are based on their experience, but you have no idea what the totality of that experience is.
You mention you like jazz. The other person crosses their arms because they stubbed their toe in the morning and jazz was playing in the background, now they remember their toe hurts and they're feeling shitty. How can you possibly foresee that? That sort of random stuff comes up in conversation all the time. The proper response isn't to try to mind read or to convince them that jazz is wonderful.
It's to talk about something else.
I may have just crossed my arms, but if you'll look closely you'll notice I wasn't looking at you when I did so. It's not that you said something that I'm closing myself off from, it's that it sent my mind on a tangent that made me feel insecure. There's a number of reasons to cross one's arms other than shunning away from people around you.