This is not really true. There are plenty of people that are retired with ~$1MM in assets and thus, are not part of the rat race. But they live a frugal life if they continue to live in America.

If you're extremely frugal, have a very small family (no kids or at most one) and live in a world where inflation doesn't exist, $1M would last you (and I'm being VERY generous) for about 20 years. If you're 40 now then your future will be $0 at 60, which happens to be the time where the really expensive stuff kicks in (like health issues and insurance). So, nope, 1M is not enough to stop worrying about money; neither 2-3M, about 5M+ is where you can retire and coast on that if you manage it wisely.