I'm with you - I want to bank roll 20 years of doing whatever I want to do as well but I just can't find the motivation to do the day-to-day for the payoff that would come so late in life. I'm afraid the part of my life that matters (my prime years basically) most will be gone before I'm even ready to relax.
I'm not sure if it counts for your question, though. In a sense you could say that I don't aim to "be rich" in a qualitative sense, but to "be middle class for a long time without working". And you may be asking why more people don't aspire to be rich in that qualitative sense? On the other hand, a person who can live a middle-class lifestyle for an extended time without working is objectively pretty rich...