Using a second language is very cognitively demanding, and it takes a while to get over feeling stupid; that's all the more reason to climb that hill quickly, and integration (especially language) programs at a university can help. I studied abroad at an elite school in France with very little French language experience, and I would have loved to have had more social opportunities (the students were very heads-down and the class structure didn't encourage socialization as well as U.S. universities) for exactly the reasons you cited, not to mention better support for learning the language (the school had a conversation partner program, but not much else).