I think you greatly overestimate population growth rates in the past here. A world where 4 successful sons was the norm has population doubling every decade or so, no society has done that. A world with just 2 has doubling every 25 years, and this was extremely unusual in the pre-industrial world. A few places in the new world (like Quebec) managed this for a while, and it was an astonishing phenomenon to people (like Malthus) who saw the numbers.

Most places were much closer to equilibrium, with about the same number of people farming a given district from one generation to the next. Something more like 3 adult children, one of whom never has a family, is what you should imagine. And then it's easy to picture multigenerational households being common. As they were, in many (but not all) parts of the world.