The problem is (if you like game theory), that it's in everyone's personal interest to migrate somewhere richer. However, if everyone does that, wherever they came from experiences such massive drain that it'll be worse off. From a purely pragmatic standpoint, unlimited immigration is a poor 'equilibrium' for all involved. 'Developing' nations will never develop, and developed nations will face massive over-crowding.
The reason the west currently allows immigration in the numbers we do is because it's an easy way to get skilled, motivated labour - at once allowing for economic growth as well as driving down the relative cost of labour.
Freedom to travel/migrate should be a human right, however there are many problems with that, considering the current state of the world.
In the US, the most prosperous states are definitely also the most populous.