Hunter gathers actually tended to have very 'casual' lifestyles with easy access to food. Infectious disease was less of an issue due to low populations making them harder to transmit. Violence however was the major issue keeping populations down.

Farming allowed for better defense though increased population density and critically by limiting migration limited contact with outside groups. But, historically farming involved vastly more work.

However, due to are vastly longer history as hunter gathers the average person throughout history probably worked ~2-4 hours per day and had plentiful access to food.