Still, I'd avoid probably avoid using "natives", even if it was meant purely as a neutral label. "Locals" would be better - if it really was important to mention their cultural/ethnic background, I'd use whatever term they prefer for themselves.
"Locals" would mean anyone living there, though, including immigrants. "Natives" implies people who were at least born there, and possibly the long-term traditional inhabitants of a place. "Native" is not a bad word.
Humans are pretty good at seeing in the world whatever is on their mind or that they have fears or insecurities about. Sometimes that's the face of Jesus on a piece of toast, sometimes that's a scary figure that turns out to be a shadow, sometimes it's interpreting an innocuous word as reminiscent of injustices centuries in the past.