There's no real need to speculate about the lives of medieval peasants when similar comparisons could probably be made with many parts of the developing world today, in villages where most people - earning just above subsistence - spend most of the time doing very little, even when there are obvious minor home improvements they could be undertaking given sufficient motivation. As there's nothing they know of that they can do to earn more income or improve their social status in their current environment, they do spend a large proportion of their time watching TV, or playing board games, or snoozing in the shade. The leisure time is real, but so is the lack of real earning opportunity and the certainty of dying - probably prematurely - as poor as the day they were born. And if all their kids survive to adulthood, there'll be even less work to do but perhaps not enough land or other means of earning sustenance to go around...

The number of people willingly leaving those peaceful villages to work in appalling conditions for a couple of extra dollars a day in arguably futile pursuit of prosperity definitely outnumbers those moving in the opposite direction.