You’re making an assumption that immigration lead leads to a better economy, ignoring the possibility of more competition for jobs, lead leading to less power and wages for workers.
You're focusing on one side of the supply-demand graph. Immigrants increase both the demand & supply side, they require more jobs but also the increase in population increases demand for more products and services. There is no fixed amount of jobs in a country.
I do agree if a country were to let in 100 million immigrants over night or some insane number that it would mess up the economy, there is an upper bound to this.
I would also introduce the concept of culture – if the economic indicators seem fine, but the culture is becoming balkanized and sectarian, then I would argue you still have a problem driven by quantity and or pace of immigration.
How do you measure culture empirically and how can you empirically say that culture is being balkanized specifically by immigrants and not other factors? You can't just take anecdotal evidence to say okay no more immigration because you have big feelings about it. There are places in the world still where having a non-zero amount of people of a different race moving into the neighbourhood would trigger some people to make the claim that the culture is becoming balkanized and sectarian.
While it's cute to think that suddenly these jobs are going to Americans, but these are mostly undesirable jobs that pay less than typical jobs. To the extent that they even get filled by native workers, it drives real wages down and not up.
Meanwhile, only allowing immigrants from the top of the income bracket crowds out the top side of the job market.
It's a lowering tide that lowers all boats.