Yeah, that's definitely true. My brother is an applied linguistics grad student, so I've heard this before.

I've had instructors who had accents but could still speak English very well, and many people just dismissed them outright as unintelligible without even trying.

But I've also had instructors who just weren't fluent in English (mostly grad student instructors). When VanillaSwirls said they didn't speak English well, that's what I was thinking of.