For me it was learning Ruby that changed programming from something I enjoyed doing as a job, to something I also loved doing as a hobby and a passion. That was several years ago and since then I've worked on many side projects, many of which are not Ruby-related.
For the first few years, I literally did zero programming on the side. I simply didn't have the energy left. I used it all up at work.
But then a couple years ago, I started having enough energy. Now, I love to do side projects... That is, if I can find one worthy of stealing my free time. (Lately that hasn't been a problem, though.)
So I can imagine it quite readily, even for the most avid programmer.