This (Bach) is a parallel that comes up often in both lay and professional music on the nature of "genius".

I've had some success with the following analogy. Music is not math, but the conversation between research mathematicians is a bit like the conversation between composers. In order for them to understand each other well they must understand the notation, but also the instruments/tools, the history, etc. However we non-composers can still listen to what they wrote if someone performs it for us - we may not catch everything another composer does, but we can enjoy it. The tough thing about mathematics is that there is no orchestra or possibility of one; in order to consume it at all you have to learn some of the details.

Music is no exception either. Many if not most people (myself included) do not appreciate highly advanced jazz as it's 'spoken' in a language too technical for it to make sense to them.

its often said around school, that the higher level mathematicians cant sit down and talk to each other about their work much .. things go too far into specialization..

This is true, that there are lot of silos in mathematics which make communication difficult. In this sense the analogy breaks down a bit.