Isn’t a lot of cost of housing in local labour (building the houses or building the bulk materials like brick or planks locally)

Would prefab housing really help with poverty when the production is suddenly offshored?