That seems plausible. I think it's reasonable to suppose that people who grew up in large families might themselves prefer to have large families, but I also suspect that personal preferences like that aren't all that sticky across more than a few generations. Maybe it balances out. For now, we can just say that most people in well-off societies aren't opting to have large families and that doesn't appear likely to change in the near future.