Well, statistically speaking, most of them do. They get a job, family, etc.
You do not have the statistics to back that statement up.
People usually have less time for games once they have kids, but they have less time for basically everything. Super Mario Bros came out 28 years ago; it's existed for essentially the entire life of anyone younger than mid-30s, into people in their 40s for it being around since their childhood. And that still doesn't even cover people that grew up with an Atari 2600 (36 years ago) or grew up in the culture of arcades (Pong was 41 years ago).
Even getting past the "competitive level" that the GP is going on about, the comparison to spending the afternoon "watching the game" is exactly right, and it's really not that big a deal for most people.
(the best statistics you'll find, by the way, put the median age of gamers at 30--meaning the every single person in the group you think soley cares about games is matched by a person older than that--though the methodology is so-so and it includes a wide variety of "gaming", including on the ever-present mobile device)
If you sleep 7 hours a day and work 40 hours a week there's still 9 hours a day to do whatever you want and free weekends assuming you work a typical 9-5 job which might be common but certainly isn't what everyone does.
How you choose to spend those remaining 9-10 hours a day is up to you. Some people would rather watch TV for 3 hours a night and sleep more so they lose almost half their free time because they enjoy sleeping in and watching some shows. Nothing wrong with that but not everyone spends their time doing that.
Look at how many grown men/women play or watch televised sports. There is nothing different than some 40 year old person watching an NFL game and a 40 year old person watching a quakelive or heroes of newerth tournament.
I don't watch regular sports but I can see the appeal. I mean, look at a sport like basketball. It seems like a single player can almost carry a team by himself as long as the other players are somewhat competent. Watching someone who is so good that he makes "professionals" seem like amateurs is really entertaining.