A fairly well (but far from universally) accepted thesis is that war and violence was widespread in pre-civilized times, keeping the population density low more "efficiently" than cycles of over-expansion and famine..

Pre Holocene, there was just more climate variability too.

I mean maybe we could have spent more time hunting, gathering, and birthing, but I don't begrudge the ancestors for just kicking back instead!

I don't get it. You're saying there were a bunch of happy, well fed people back then, just kicking back and relaxing.... and not having sex?

I don't fully get it either. Certainly the agricultural revolution led to more babies and more calories (if not better nutrition in the early days), so it is possible to the calories / babies ratio stayed the same.

For the hunt gather nice times stuff to work out, I think theoretically there must also be some claims about the work being divided more evenly. Among other things, that could be some restrictions on how much pregnancy the nomadic band can deal with.