I guess it's nice to hear I'm not alone. Well unfortunately you need to fill in the other 4 hours in a way that doesn't leave you feeling inadequate - but you're right, it's something to work on.
If I had it my way, I'd work half time programming and half time working a public facing job.
Sometimes people claim they can be productive for 8 or more hours per day, for weeks on end. But likely what's happening is they are tricking themselves into feeling (and looking) productive, while their actual productivity finds an equilibrium far below what it could be if they paced themselves.