It's true that rent seekers who currently use state-sanctioned violence to prevent economic competition might be harmed in a visible manner.

But factory workers are consumers too and gain (less visible) benefits from immigrants. I.e. if you work at factory X but consume goods produced at factory Y, you benefit from cheaper labor at factory Y. If you work at factory X but consume medicine, you benefit from having an Indian doctor and a Filipina nurse.