There is a incorrect assumption that people often make, without malice, about economics being a zero sum game. It really is not, almost by definition.

The more people in an area, usually the more opportunity people have for employment over a long time period, and the higher the economic output.

Also, there is a difference between refugees (i.e. people who might otherwise die) and economic immigrants. It's hard to fault someone for their perception that other people might take their jobs, correct or not. It is easier to fault someone for taking the same stance when the alternative for the immigrants is potentially far more catastrophic. Especially when the arguments are more about culture and race than employment.