That argument is extremely broad, though. It's easy to live off your investments if you live in a hovel, eat ramen every day and never go out. Is that classified as rich?
Sure, it's a continuum, but "not having to live hand-to-mouth" is a hell of a lot closer to rich than to poor.
You still have to go to work every day like everyone else to sustain this lifestyle. You just have nicer "things" and "fancier" experiences?