The hypothesis I've heard that supports this point of view is that your language center is single threaded. You can either comprehend in your native language or you can comprehend in a different language, but not both. That said, the value bindings are arbitrary. So when you are naturally a single language person (aka an American [1]) that language binding monopolizes the language center. And trying to go through it to learn a new language is like trying to learn to program in C by using the C pre-processor. Everyone I know though who knows 3 or more languages has said that after 3 its much easier. I have heard people compare to learning "binding arbitrary sounds to language" rather than learning a new language.

[1] Old joke which goes - "What do you call people who speak two languages? Bilingual! Three languages? Trilingual! One language? American!"