Modern day programming is so much more than just opening up your IDE and churning out code. You have to collaborate, talk, review other people's work etc. There is also a meta aspect to it to improve you productivity at work by looking at your own work from a level above.

You also have to keep learning new bleeding edge stuff to not be out of business in a few years.

Plus doing a lot of side projects and extra work, always opens up avenues for new opportunities and helps you to meet with other smart people who often have something new for you.

All of this is work. And there is no way you can be good at this if you only work for 4 hrs a day.