Instead let's try a hypothetical. Perhaps someone has a historically oppressed indigenous background (perhaps they even have brown skin) it is possible that they may not want any more immigration into their lands.
In my opinion that would be a very understandable position for them to take given the historical circumstances.
The economic argument isn't as clear cut as you suggest either. An economy with excess supply of underutilized labour isn't going to see many gains from immigration.. There's market distortion to consider with regard to housing supply as well, and plenty of other reasons someone may take a valid position against immigration without being racist
https://britishonlinearchives.com/posts/category/articles/62...
https://asiatimes.com/2019/06/malaysias-indians-face-growing...
https://www.latimes.com/archives/la-xpm-2000-jun-22-mn-43757...
It's racist for people born in or who are citizens of a country - today - to resist the governing class allowing foreigners to immigrate to benefit the economy AND it's racist policy - historically - for a governing class to push immigration on a natively born population for economic reasons.