From my musician friends too, I seldom see them ever spending more than 8 hours a day doing music. As music work isn't really 9 to 5 style.
Here we're talking having done an 8 to 10 hour day programming, Monday through Friday, and then spending even more once home and on weekends.
Now, I happen to be, maybe as you are, someone who is passionate about CS. So I do actually do it as a hobby and as work. I just don't believe everyone should, and I don't want to make it the norm that you have too, because if that was the case, my doing it as a hobby for myself would now become me doing it cause I have too for employability.
It's also not my only hobby, I found that when it was, it did in fact slowly overwork my mind. I'd slowly start sleeping later, waking earlier, having issues falling asleep because I'd always be thinking about my problems. A few hours before bed, switching to a non thinking hobby has been really nice for me.
I agree with you about the stigma though. I think if anything, I'd like to encourage non-programmers to pick up programming as a hobby. It's a great creative outlet, and the computer is a tool. It can be very rewarding and so much fun. Plus, for many, it could translate into a better career for them.
As a programmer though, I think it's just up to you, but definitely you shouldn't have too.
Just my opinion.
The programming "interface" is more or less staring at text on a monitor and clicking on buttons. All the creative work is hidden and the interface is complete crap... How can that be perceived as fun or creative by someone looking from the outside?
Music making always had an aura of coolness, so even if a professional musician were to do that, maybe people would not see it as work.
The computer is a tool. It's the tool that I use to create things, and that creativity gives me life. I don't think people without creative hobbies can really understand this.