And then there are languages entirely without any concept of grammatical gender or gendered pronouns, such as my native Hungarian. So the controversy doesn't carry over easily to Hungary, at least not in the same form. Of course someone may explicitly use words like "woman" or "boy" in the unpreferred way, but you don't have spend extra effort to be neutral/ambiguous. This also makes translation quite difficult, when a writer relies a lot on he/she to mark who is speaking in a conversation between a man and a woman. In such cases the translator has to sprinkle in the names to better keep track of who's speaking.
This can affect you even when you're aware of it. I've spent a lot of time with non-native speakers and I've been the non-native speaker in other scenarios. I've gotten close with a non-native English speaker who recently moved to the states for engineering. I forgot how the language barrier can unconsciously affect me. I knew he was at least smart enough to be an engineer but just this past weekend he told me more about his life and career. He was modest but blunt and straightforward about his accomplishments, in my own words I'll say it's prolific. It was a good reminder for me. Unfortunately most of the exposure I have these days with people in the category of "struggling to communicate" are children. What a terrible mistake it is to subconsciously associate non-native speakers with the intelligence of a child.
I guess it wouldn't annoy me so much if my job and social circles (many of them involving foreigners who don't speak my native language) didn't require me to consume so much of it. There's so much subtlety, nuance and "color" that's lost in a conversation where both participants aren't speaking in their native language. Even if both are relatively good at it.
Then there's the issue of native English speakers who seem to be only very dimly aware that other languages exist and think that everyone speaking their language fluently is a given. (This shapes a lot of current politics, such as language policing or the whole 'pronouns' issue - I'm mentioning it because antirez was the center of a controversy for not changing terminology that's loaded for Americans only). And many native speakers think the less of people who struggle with English, even unconsciously.