It's possible to resent someone who shows up and works hard. (Harder than you'd like to, because he's more desperate.) It's much easier to resent someone who shows up and simply collects a slice of your inheritance.
And in addition: The guy who stepped off the boat in 1900 and then spent a few decades working long hours in a factory, spent a lot of time interacting with other people in his new country. It was unavoidable. (Possibly he got drafted, too, another great way to spend time with the locals.) One of the reasons integration is not going so well in Europe is that this is no longer the story.
Not sure if I agree with the xenophobia claim, although perhaps you've seen studies or statistics that I haven't.
Do you think substandard living conditions will cease to exist the moment we remove relatively high paying factory jobs and external revenue from impoverished countries?
1) Human labor will no longer dominate COGS--global shipping and logistics will. There will therefore be a massive onshoring of manufacturing back to the developed world. As a result, we will stop subsidizing human rights abuses and substandard living conditions around the globe.
2) In developed countries, this massive onshoring of automated production will make it clear that low-skill, high-paying factory jobs are forever a thing of the past. Simultaneously, xenophobia--"they took our jobs"--may cease to be an effective political tool. Perhaps this will trigger an honest discussion around UBI and revamped social safety nets.