You're discounting the fact that most extant native peoples almost certainly displaced other native people by the time the Europeans showed up. To provide a single example, Southern Athabaskans ventured south from Canada and displaced the Pueblo people after Europeans had already started showing up in North America.

Also, "aslyum-seeker" and "immigrant" are two distinct concepts.