If it's not displacing paid work, then it becomes a matter of subjective valuation of your time, which depends not only on the time, but the task and what you would do instead. In that case, you really have to place a subjective value on how much you like or dislike mowing the lawn, and how much you like or dislike what you would do instead. Since you're just displacing one kind of unpaid time for another kind of unpaid time, there's no objective economic basis for determining what the delta value between "sitting in a chair" and "mowing the lawn" is. I personally put it fairly low, because I don't mind mowing the lawn (plus it's exercise, something I'd have to find time to do anyway).
Of course, the question then is how do you value that time? I don't think there's any objective answer to that, but it's probably more than zero.
EDIT: Of course, I just re-read your comment and see this is basically what you just said. I'll admit I got distracted and misunderstood your point the first time I read it.
This is an emotional response to the economic mindset that the article purposes. In the article - the entire point is that it doesn't matter what you do with that extra time, but if you value your time at $100/hour and you do a task that you can pay someone to do for $10/hour, it's not an efficient use of your resources. That's the economic theory - whether or not you do anything with that extra time is up to you.