My experience is different. I'm not a native speaker and if I listen to music in English, usually it just fades into background. I rarely experience it. In that way I feel it does not have any effect on me, but obviously it's anecdotal and I don't have any real data.