The most readily noticeable part of the adult barrier to language learning, which I have run into more than a dozen times as an adult learner of a variety of languages, is acquiring sufficient phonological proficiency to speak understandably and listen with good comprehension. That's because production of the phonemes (and perception of the phonemes) of your native language(s) is based on ARBITRARY grouping of sound distinctions that are grouped differently, or entirely ignored, in other languages. Each language has its own sound system, and generally any learner who begins study of a language after puberty (especially if male) will have a noticeable foreign accent.
http://books.google.com/books?id=r2HEqsWeQTYC&pg=PA30...
But there are also more subtle problems with second-language acquisition of vocabulary and grammar. Each language groups concepts with words in a different way from any other language, and each language has grammatical rules that are wholly arbitrary from the point of view of any other language. But usually second-language speakers and writers can bluff better as to vocabulary and grammar than they can as to pronunciation.
I've often worked as a Chinese-English interpreter. I didn't speak any Chinese until age seventeen, definitely after puberty for me, but I have acquired a decent accent, not such a thick American accent that I can't occasionally pass a Chinese person over the telephone--but mostly as a southern-Chinese second-language speaker of Mandarin to other southern-Chinese second-language speakers of Mandarin. The issue of "lexicalization" (really knowing the correct word in the acquired language) really never goes away. And there are whole thick books about Chinese grammar that few Westerners have read cover to cover.
All the time and effort needed to learn a second language has given me much incentive to encourage my children to grow up as bilingual persons--but even at that it's hard to find an adequately bilingual environment to support daily use of both Mandarin and English.
I learned to speak English when I was 10, and I have no issue with the differences between Chinese and English. But I was exposed to French only a little bit later (12) and I am completely hopeless at it. The grammar, spelling and vocabulary are the easiest parts; I just can't seem to produce the same sounds.
I wonder if babies and young children can acquire language quicker simply because their brain is optimally tuned for maximum language learning (e.g. if they hear a word they might not just be thinking of the meaning but the emotional context, what it means to them, improving recall or if they learn a word, they intuitively understand the concept behind the word because they learned the world for the first-time in the real context [e.g. 'fork' in kitchen]).
I dream about a virtual reality add-on that would let you roam outside in the real-world and it would augment your reality - through pop-ups of the Spanish/French whatever word for objects that you see. As one of my Spanish tutors told me, the key to learning vocabulary is to be able to associate the concept of the word with the word (in whichever language).