So it is a significant trade off.
Early retirement is almost always an option for middle-class or better Americans. It's just a question of how much pain you want to put up with in terms of housing, lifestyle, where you want to live, whether you want great education for your kids, etc.
I used to agree with your comment but don't anymore. I have enough money at this point to earn 10s of K/year investing in real estate or doing something else. I don't because I like working in tech, being part of a team, using my education and talents, and generally working on making the world a better place.
I don't really want to "do whatever I want".
These guys are right: https://signalvnoise.com/posts/1930-mojito-island-is-a-mirag...
Your middle-class also seems to be a bit different than my middle-class, especially in cities where jobs are.
If you're making 200k on your own terms vs 250 at Google, I wonder who's better off. At that level of income, things like how stressful your work is, how interested in it you are, the length of your commute, etc. seem to matter more than earning another 50k of income.
Obviously everyone has their own circumstances but I think the obsession with earning as much cash salary as possible seems a little myopic.