It's about supply and demand.

If you supply labor at X skill level, an influx of immigrants at X skill level (but not of Y or Z skill level) will reduce the buying power of your wages.

Taking it to real life - people tend to oppose immigration when they are at X skill level, and support it when they are at Y or Z skill level. It's raw self-interest often dressed up as altruism or principle.

You can see this understanding here whenever H1B is discussed - suddenly supply and demand becomes an obvious economic concept.