Early retirement is almost always an option for middle-class or better Americans. It's just a question of how much pain you want to put up with in terms of housing, lifestyle, where you want to live, whether you want great education for your kids, etc.
I used to agree with your comment but don't anymore. I have enough money at this point to earn 10s of K/year investing in real estate or doing something else. I don't because I like working in tech, being part of a team, using my education and talents, and generally working on making the world a better place.
I don't really want to "do whatever I want".
These guys are right: https://signalvnoise.com/posts/1930-mojito-island-is-a-mirag...
Your middle-class also seems to be a bit different than my middle-class, especially in cities where jobs are.
So it is a significant trade off.