Libertarians do make exactly that argument about immigration. And people who disagree make exactly the same counter-argument - that even if it is a net benefit globally, some people might not like the result locally.