There was always work to done. If wasn't something that was pressing right now (fieldwork, baling hay), then it was needed to be done soon (fixing fences, equipment maintenance), or improving your infrastructure (building better winter pens for animals, better storage for hay/crops). Add to that without IC engines, all work has to be done using human or animal labor. A Speaking of animals, they don't take days off. If you have milk cows, that's 20 minutes a day (for each cow) of milking by hand. Think your commute sucks? Try walking everywhere you go.
Just go to an Amish community and you will quickly see that they spend more than 40 hours a week working, and most communities still take advantage of some technology (small tractors, tools and other items that are manufactured by the outside world).
There seems to be this idea among office workers that the M-F, 9-5 "grind" of cube-life are the most terrible conditions humans have ever lived through, and it shows a laughably sheltered viewpoint.
It's morning to evening back-breaking physical work, undernourishment as a matter of fact, and very high probability of death - some of the families we talked to had been 2/3 decimated by the age of 35, from traffic accidents, a host of diseases like cholera, snakes, you name it... There is no question when I'd rather live.
I've lived in third world, "second" world, and a host of first world countries including the very desirable Australia, Singapore and Switzerland and I've definitely found a strong correlation between wealth (including scientific and political) and happiness.
Past a certain level of personal wealth and regardless of the surroundings (although they affect that level), philosophy becomes more important, but that's another argument. Even there, I'd argue more prosperous countries are correlated with sounder philosophies and happier HNWIs, but my sample size is much smaller.
A medieval peasant most likely had more leisure time than you. Having a local technological maxima doesn't mean "constant progress".