In a making stuff up sense, you can fit near any narrative you want to near any set of data about our history. Consider, used to be there was literally nothing to do for the majority of the day/night for folks. Yes, you could tend the garden and such, but that was done before sun was fully overhead or you would suffer exposure symptoms. You could watch the herds, but that was mostly passive, for mostly the same reasons.

With that narrative, many scientists in history were not people that didn't have to do some of these boring things. They were people that filled their free time with experimentation and other observations.

Of course, that is just one narrative to fit to the data. I'm sure there are others. But it is not just incumbent on one side to give evidence. Can you point to evidence that free time is not beneficial? Observationally, as a parent, forcing my kids to have boring free time has led to them doing/learning as much as forcing them to productive learning time. (Indeed, I've grown sympathetic to the argument that productive learning time often isn't either of its claims. )