Which is already the vast majority of immigration.

everything else would just be perceived as "taking someone's job"

In a world where there were only a fixed number of jobs, family members immigrating would also 'take jobs', so if people really believe that, they should aim to keep out family members too, since they are way more of them than people in the H1B program. For that matter, they should also get nervous about people who have lots of children, as those children will (in theory) grow up to take more than their parents 2 jobs.

Of course there are not a fixed number of jobs, and keeping families apart is generally viewed as cruel.

thats the thing about immigration. everyone wants the door shut behind them, even recent immigrants.