This isn't about not wanting immigrants.
This is about not creating false reasons for which we claim immigration is necessary or important.
That's it. Simple as that.
> the more great people we have, the more innovation we will have
Absolutely, positively, unequivocally false.
Innovation is NOT a function of quantity.
General Motors has over 200,000 employees.
Tesla had about 900 employees in 2010. They innovated the shit out of a market dominated by giants. And that's the story of every single small team that has disrupted market segments. Read "The Innovator's Dilemma".
More people does not mean more innovation. This is demonstrably false across a wide range of industries.
We do not need to import people to innovate and foster entrepreneurial drive. We have enough smart people already on board.
Sorry that this argument doesn't work for you. But yes, you are engaging in a variant of the "lump of labor" fallacy: the more great people we have, the more innovation we will have.