It would also decrease prices and thus raise living standards. Customers would then turn their attention to better goods and services, increasing the demand for labor. You're committing the lump of labor fallacy.

Think about it this way, if two countries were to merge, would everyone be worse off because of the increased competition on the job market? No, people would be better off due to the increased amount of trade.