It's a mix really, peaks of heavy physical work, usually during the summer, lots of downtime in the winter, with mostly animal feeding/milking twice a day. This lifestyle is still somewhat prevalent in Eastern Europe, and one can easily tell apart villagers who work the land and those who work in factories, the latter being more stressed and worn out both physically and mentally. (source: personal observation across several decades)
Pure arable farming is a little different, but harvest time would be dawn-to-dusk working for everyone.