The non-tech job I had for one Summer is probably my favorite job I've ever had, pay aside. Paid minimum wage or just over.
Ha, yes, concurs with one of the aspects of my work dissatisfactions also. The sheer dreary meaninglessness (or worse) of nearly everything I've been paid to do. It's pretty wearing over time.
Rather than mental or emotional fatigue, more of a problem can be injury, or accelerated decline in fitness to do the physical work after labouring for years; while still needing the job for money.
Anecdotally, every single fit, strong & able old person I've known has been a manual worker. Case in point: 90 year lifelong builder currently doing roofing work. 87 year old (my sailing instructor!) currently running (on the ground) his tree surgery business. Most lifelong office workers I know are barrel shaped and essentially disabled by their 50s.
The stats might be interesting, though I suspect they'd be pretty complex, different by region & nation, and highly confounded (by class, lifestyle, etc).
But I have truly loathed every microsecond of every white collar job I later took on, for a variety of reasons. I'm relatively immune to envy on the whole, but I do admit to it for people who have found satisfaction in work (not that I have known many).