I'm not suggesting that people only read fiction for self-improvement or to make you better at conversation: there's nothing wrong with reading something for it's own sake. Is reading something for its own sake better than watching a movie or playing a video game for its own sake? Probably not.
One way to get over this feeling may be to keep in mind that the raw accumulation of knowledge is not really that valuable anymore - certainly less valuable than mental "agility" (for lack of a better term). In other words, we can look up any facts we need to know pretty easily, but being able to put order to those facts and make correct predictions based on them is a skill that takes practice. I think that some fiction may be better at fostering that skill than most non fiction - especially labyrinthine stuff like Neal Stephenson, Tolstoy, or Thomas Pynchon.
But this brings up another question: why does there seem to be this general obsession with the consumption of this particular medium? I would say they're in love with the form, but they don't mention doing any writing. I would say they're in love with great art, but they don't mention any other forms. (Is it a NYC thing? Manhattan always seems downright foreign to me.)
Imagine asking the people quoted in this NYT article about the last manga they've read or the last modern symphony they heard or the last painting they made.
By the way, I mostly agree that books are overrated. The reason is that most books are not very complex. Text is the wrong medium for the ideas of their authors.