ESL speaker from Germany here. My experience is that I can cope really well with the accents that I understand, but I struggle really hard with new accents until I crack their code, so to say.

For example, the wife of a friend is from Turkey and is not very proficient in German, so we talk in English. The first few times it was really tough for me to understand anything, until I figured out how the Turkish accent works. For example, the Turkish accent replaces th -> t, e.g. "tank you" instead of "thank you", which was really confusing for me since I'm used to German ESL speakers who replace th -> s instead.