This isn't good reasoning. Each immigrant still has own motives and his own thoughts. A large number doesn't render these individual immigrants as a part of collective formless blob or a de facto invading army.
Consider that not all immigrants leave their home countries because of their governments. Many leave because of religious/cultural persecution from one's local community (e.g. homosexuals) or unacknowledged rights (e.g. abortion) in an otherwise secular and ostensibly stable country.
I think you're misreading Nozick. Nozick would suggest open borders, extenuating circumstances notwithstanding. The argument for closed borders would imply that any authority over a country's or state's territorial holdings would belong to the nation's government rather than its people.