Even in the case of migration to avoid a bad situation, it's still a small minority that left whatever the old place was. The vast majority stayed there, so you've still got a tiny self-selecting population of immigrants.