That argument is extremely broad, though. It's easy to live off your investments if you live in a hovel, eat ramen every day and never go out. Is that classified as rich?

Sure, it's a continuum, but "not having to live hand-to-mouth" is a hell of a lot closer to rich than to poor.