Because it is a dangerous fallacy to presume that all peoples are simply replaceable. When you import people you import culture, and that potentially leads to friction.