As a non-native speaker, i rarely have an issue with advanced vocabulary or idioms from native speakers. The thing that is hard to me is understand heavy accents of other non-native speakers or local accents.

Also some ungrammatical constructions are hard to understand, e.g. when i first saw sentence with "your" instead of "you're" it took about minute to decode it.