I currently work with people¹ who quit construction & plumbing to retrain in tech for pretty much the reasons you describe. The two I work with right now are a dev manager and a support tech, another I'm no longer in contact with was a senior dev, none of them to my knowledge hanker to return to their old working lives.

None of them were top-level contractors, those who plan and hand out work to others, instead being people on the ground working on sometimes physically demanding tasks in all conditions and being the public face of the operation so sometimes interacting with difficult members of the public. Maybe if you get one of those management/middle-management jobs in construction the dynamic would be different, but then again you'll have a lot of the same problems that make me avoid management/middle-management jobs in our industry.

[1] and have known others