Exact same thing in South Korea. Folks afraid of embarrassing themselves with their English, but once you speak Korean (and look presentable) it's not a problem.

Funny thing about the oversensitivity to racism: many foreigners assume racism, but if they understood Korean they'd figure out that either they're not the topic of discussion, or the locals are gossiping or speaking the same way they would about a local Korean person as well.