Here's the thing, all that other stuff you mentioned are things I can do with my family. Side projects aren't. I have trouble reading seriously now for the same reason; it is a solitary activity. But at least if I'm reading a book I can still pop out and participate in conversations; not so with programming.

The other important difference, and the reason there is always strong pushback when this topic comes up, is that many of us believe there is a subtext that people who spend a lot of time outside work on side projects are better employees. (I mean, it isn't even really subtext, it is in the text of the OP, and I've been reading the new edition of The Pragmatic Programmer, and it's in the text there as well.) If that view wins out (and in many circles it already has), then what you see as harmless, fun, and enriching side projects are actually a job requirement, one that is extremely hard for many of us to satisfy. So we (reasonably, I think!) push back on this view because we think we're good programmers and good employees without needing to spend our evenings doing solitary work away from our friends and families.