In mental labor you get fat and achy sitting at a desk all day, your sleep schedule gets messed up looking at artificial light constantly, your mental health degrades and your risk of suicide increases - obviously it's also unpleasant.

We can all agree the manual laborer's job is hard because no one can really escape manual labor. Even if your profession doesn't require you to lift a finger, other things in life like taking care of a home expose us to plenty of comparable tasks, and we can easily imagine how much worse it would be to do these tasks all the time. But the manual laborer can forego most of the negatives of office work which require prolonged and continuous exposure to things we would never experience in nature. Thus the manual laborer tends to see greener grass on the other side while the office worker does not.