Actually in my experience the people who learn the language form a double distribution: those who are more educated/successful and those who have nothing but the will to survive. It happens in Spain, where poor foreigners know Spanish, highly educated ones also know Spanish, and the middle group can say some words or don't bother at all.

In Japan the main fear comes probably from China, since it's a lot easier for Chinese people to learn Japanese (similar characters) than for occidental people, and there is also a "relatively" high inmigration from poor Chinese people.