This is kinda of where I am right now. I still enjoy programming, but its becoming more and more apparent that I should have kept it as a hobby and pursued something else as a career.
That's the thing - this is much easier to do when you're financially independent. If you have kids, and have to put them through school, and maybe have sick parents to care for - costs can add up fairly quickly; and it's easy to find yourself in a position where you can't lose your job, which is a lousy position to be in.
Interesting part is - it can also be easier to choose what you work for when you have more money (or at least that's been my personal experience so far). A different version of the "rich getting richer" dynamic, basically - it's easy to say "no" to stupid things, to avoid or leave crappy jobs. Also if you're highly paid that often means you're listened to/ your voice has some weight - so chances of working on stuff that delights you increase significantly.
(NSFW:Language)