So you're essentially suggesting deferred philanthropy. I totally agree with that philosophy. This is why I'm also doing a startup and not joining the peace corp. But, once you've "made it" and have more money than you know what to do with (the question at hand), I think it's reasonable to consider Philantropy and not just "fun" stuff.
An example of comparative advantage: it doesn't make sense for a $500/hr lawyer to do pro-bono work for the poor. It would make more sense to work for that hour and use the income to fund 10 $50/hr lawyers. The benefit would be much higher for those in need.
So my point is that charity isn't necessarily the best way to give back to society.