I don't know if this is 100% true. Part of the anti-immigrant rhetoric is "they're coming here and taking our jobs", which applies just as much to white collar professions as to blue collar ones. Think of all the people in tech who complain about immigrants using H1B visas taking programmer jobs. If you have a zero-sum worldview about how the economy works and how jobs are created, then all immigration is bad, and immigration of people competing with you in the same segment of the labor market is the worst.