Historically labour voting workers have been displaced from their jobs by low skilled immigrants. Don't just take my words for it:

https://medium.com/mosquito-ridge/brexit-wins-an-illusion-di...

"Britain has voted to leave the EU. The reason? A large section of the working class, concentrated in towns and cities that have been quietly devastated by free-market economics, decided they’d had enough.

[...]

Neither the political centre or the pro-remain left was able to explain how to offset the negative economic impact of low-skilled migration in conditions of (a) guaranteed free movement (b) permanent stagnation in Europe and (c) austerity in Britain. "

Were you think physically displaced, rather than "taking over the role of" displaced?