It would reduce the supply of highly skill labour, which would reduce the amount of goods/services produced, and as a consequence, the amount of wealth the world gets to consume. You're making the 'lump of labour fallacy' in assuming that someone will just take his stop with no loss.