> but consume goods produced at factory Y, you benefit from cheaper labor at factory Y

Assuming the savings are passed on to the consumer.

I think it's more important that the owners vastly benefit, and may manage to push down wages across more than just one industry by leveraging economic differences between nations that only they can take advantage of.