I used to be big into Magic: the Gathering and MMORPGs and Starcraft as a teenager. I'm not really, now; I'll play Starcraft with buddies once every couple months but it's not part of my normal routine. But I don't regret the time spent mastering game mechanics in my adolescence. That was who I was then; it's not who I am now, but it played an essential part in shaping who I am now. (Particularly since my first major programming project, like many people, was trying to build a MUD, and I taught myself C, C++, BSD sockets, Java, and OOP all for it...which turned out to be rather important when the dot-com boom hit a couple years later.)
As I got older, I was exposed to more and more amazing and inspiring things in the world, which motivated me to spend my time seeking out new and cool things to do and places to explore. My time was way too limited and important to waste doing things that wouldn't expand my horizons in some way.
As I got older than that, I realized that I was killing myself by never allowing myself to just relax. Work shouldn't be 100% optimized and neither should life. It takes a toll. It might take 20-30 years, but it does take a toll. Now I try to always have some amount of time every week (every day even) where I don't have to always be doing something fulfilling or enriching. I just do whatever I feel like doing during these times, which is sometimes constructive and sometimes not. Point being, I don't worry about it. These times allow me to recharge and rejuvenate before diving into whatever epic activity I've convinced myself I have to do because life is too short, etc. etc. etc.
TL;DR: appreciating the finer things in life is important, but learning to appreciate both the finer and coarser things is Nirvana.