People don't freak out about the numbers of immigration, they freak out about what kind of immigration is happening. Me moving on the other side of the continent and a different branch of PIE language family is non-event. The person from the country next after mine and the one after are rising all the eyebrows.