I have this problem. What I've found is that even if I am not actually doing a task, the simple fact that it exists and I've not done it niggles in the back of my brain and stresses me out.

Last night I finally cleared off all my emails and did all the little task that have been sitting around. It feels so good to not have them lingering on the edge of my brain.