People who do not have to work work much less than people who do have to.

I don't think that's particularly good evidence for that. You're confusing "has a job" with "is working" and you are ignoring the age effects that are the whole reason retirement exists in the first place.