I think some of this is because other people are working on your day off. Like if you're awaiting feedback (or code review), you find yourself aiming to enable all those people you're waiting on to do that stuff on your day off.

If they also worked 4 days a week, I bet productivity would fall.