I used to have a similar feeling about not "improving" enough, until I realised something: doing all that messing about (watching TV, films, reading books) makes you more human.

When I'm not at a local meet-up, or conference, most of my conversation revolves around non-programming stuff. Most people don't want to hear my opinions on Python, Django, Scala, JavaScript etc. but they do want to know if I saw that ludicrous display last night.

If you want to think of it in "self-improvement" terms, then imagine you're working on your personal database of trivia that you can talk about.