Seasonal farmers in Europe and Russia spent half the year doing nothing. Even if they had to work their asses off during the growing season, they had nothing BUT leisure time during the winter. http://www.nytimes.com/2007/11/25/opinion/25robb.html
I think one problem with pre-agricultural life was that it was so unpredictable. You had plenty of food, until you didn't.
Another downside was what hunter-gatherers had to do to their parents when they got too old to travel, and to children born too close together.
Think what that implies. If the hunter-gatherer mode of life couldn't support a large population, how was that limit enforced? It had to be by people dying.
But that wasn't what I was asking anyway. I wasn't asking why agriculture won, but why it started. The first agriculturists must have switched from being hunter gatherers. Why, if life wasn't better?
(I realize the switch was a matter of degree, so consider the preceding question as an abbreviation for asking why about each degree of switch.)
Re predictability: You could say that about agriculture too. Think about the great potato famine in Ireland. Agriculture's not immune to famines from droughts or pests.