I'd probably feel that way too. But I have to ask - how much thought have you really given it? I vaguely remember reading a surprising number of posts from HNW people that regretted not trying certain hobbies / activities when they were young enough fully enjoy them. That saying that no one on their deathbed says their biggest regret is that they didn't work harder, is probably a trope for a reason.