I do have the advantage of having grown up speaking two dialects (but they are not THAT different) so I am probably better than average at understanding spoken English. I think the biggest difference is attitude: I am not thrown by hearing an unfamiliar idiom.

I have heard Singaporeans, Indians and others speaking to each other and had no difficulty understanding them.

A problem I have come across in South Asia is people mixing languages. At that point are the speaking English? It can be quite disconcerting when someone changes language mid-conversation.

I have found Geordies and some Scots difficult to understand, but I think "mutually unintelligible" over stating it. There are reasons why Scots is sometimes classified as a different language. Other than that I have had difficulty with any form of British English.