From my experience, even people used to variation still struggle when it is the wrong kind of variation. It's a bit difficult to explain but I have seen it happening regularly, including once when I tried to say something about Vancouver and it took a writing down to get it across. The other person was used to one southern American accent, Estuary English and European Globish at least.

I know I shouldn't but I still do find it surprising how someone speaking English can't understand a well spoken English person!