There's a big difference between being anti-immigration and anti-immigrant. All of the latter people are also the former, of course. But there are people who want lower immigration even though they have no problem with the immigrants who are here. Some union people are surely in that bucket just for "reduce job competition" reasons.