"if things were basically opened up between wealthy countries"
so, only between wealthy countries?
For starters. You couldn't realistically completely open, Holland, say, to immigration from India and China without some adverse consequences. Between rich countries, people would likely flow back and forth enough to balance things out.
I don't think there would be any huge effects if things were basically opened up between wealthy countries... a few would go here from there, and vice versa, and things would balance out. But it is increasingly difficult. Bleagh!