(I'm currently at a crossword puzzle tournament in a foreign language in a foreign country, when I could of course find many such tournaments in my own country... so I guess I've been spending the whole weekend enjoying an experience of appreciating a foreign language in a way that's in some tension with what you describe.)
I didn't understand how to use "whom" in English until I learned (some) German, and I feel that I understand more nuance and subtlety in my own tongue as a result. Same goes for the (sadly nearly dead) subjunctive form.
Language appears to be the tool we use not just to communicate, but to encode our thoughts. It gives us a way to wrangle the abstract from something ethereal and unmanageable into a conceivable thing we can handle. Having many different languages yields many different ways of thinking, not just communicating, and we are impoverished for their death.
This is a fascinating story about the experience of a deaf person who had no language until their 20's. It affected their thinking in an utterly profound fashion - not just their communication.
I agree, the language barrier means a lot is lost. But at the same time each story and even way of thinking is flavored by the language it is expressed in. Certain things cannot be fully expressed in another language, the words don't hold the same charge and history.
If the entire world were to speak one language tomorrow and nothing else, there would be a great diminishment in the world's art, culture and mythos.
So it's not a black and white issue.