That idea of productivity is odd to me. If I build a house for you and you build a house for me and we both compensate each other with the same $ amount for the work, is that a net productivity of 0? Two new houses got build.

you're right in that regard. But I'm looking at it as net of the whole system inputs.

You put in one house, I put in one house. There are no resultant houses for others to consume remaining.

Another way to think about it might be manual farm labor. If an acre of food grows enough potatoes to feed 1.2 people (idk if that's true), and it takes 1 person to labor over. The acre is only producing enough food to have .2 people as non-farmers.

If there are only non-farmers, the farmland doesn't produce any food (discounting wild berries or whatever grows when farmland is wasted). The farm can simply produce for 1.2 people...

That's 25 single family homes for company owners only. Don't forget they also gave jobs and salaries for a thousand of people, and a product for the market to use.