(Modern USA infant mortality is less than 0.6%)
Death is a part of life. Everyone hopes that they outlive all their children, but even if you do not, tragedies during your life do not mean that your entire life was unhappy.
Also, how do you weight the absence of toys and creature comforts?
Your point about people being sick is valid, but not to the extent that you think. Firstly, it is not wise to suggest that hunter-gatherers were physically worse off than us. They got a lot of exercise, and were in all likelihood better physically. Secondly, the shorter lifespans statistic is an artifact of increased childhood mortality. Once you discount the child mortality, they had lifespans very similar to ours. Without all the diseases of affluence that we have, mind you (diabetes, for example).
I think it's important to look at war, disease, hunger, and displacement, in some order.
Back in the time of hunter gatherers, humans "worked" for a few hours a day. The rest of the time was spent socializing and in leisure. It is fair to say they were happier compared to, let's say, an African American debt-laden minimum wage worker in the US today.
Do we really need economic growth?