How does having poor people living and working in the same country as you make you poorer? Your ideology looks to me like socialism except in reverse: you want to restrict free capitalism in order to keep the poor segregated and in poverty.
Phasing it in at 10% increase in total immigrants per year would be fine as well.
Many of the people living in shanty towns are victims of failed institutions, corrupt leaders, and oppression. With this removed their resulting standard of living would likely be at least middle class. The lazy and unmotivated from those nations would stay there, and the motivated would arrive on US shores ready to improve things for themselves.
It's just plain osmosis. Open the borders, and people flow into the US until it reaches the point where living in the US is no longer any better than living... in whatever the current worst country in the world is.