Pre-deregulation, they couldn't compete on price, so they competed on quality of perks (free food, service, etc.). Once they deregulated, they competed on price, but then started cost cutting on perks to make earnings. It was only after LCCs and the "premium experience LCCs" were successful that airlines both eliminated bundled perks and provided for-fee superior quality products for sale.
The "economy plus" type products for frequent fliers also go a long way to make economy tolerable for larger people.
I think air travel will continue to improve -- the biggest setback has been the 9/11 security increase, but other than that, the legacy carriers seem to be getting better.
I still try to drive whenever possible.
As a somewhat tall person, I feel your pain.
Coach, as well, is certainly nothing like it used to be. It's been getting better in recent years, to your point, but only very slightly. I think it's still somewhere in the basin of an overall historic trough, though perhaps with upward direction.
(Full disclosure: as a very tall person, I feel changes in seating configuration, etc., a lot more accutely than most people do. So it's possible my opinions are either amplified by, or colored by, my individual perspective).