If you built the houses the same way as with human labor, sure. But perhaps they would be built differently with robots, in a way which didn't require that level of dexterity. After all, most industrial robots don't have opposable thumbs.

I'm not saying it's a simple problem to solve, nor one whose solution I have fully mapped out in my head. Just that it's a problem whose solution strikes me as being within the realm of feasibility.