The most impactful activities I pursued during my PhD had absolutely no bearing on my research itself.

Here's one example: I created a robotics blog where I wrote about some of the new, interesting developments in the field that piqued my interest. It ballooned into one of the top 3 robotics websites on the web. I felt guilty about it for a long time... until I realized that the blog had a bigger impact & reach than any of my research -- I was known in the community; articles were cited on Wikipedia and in Congressional testimonies; and it established my credibility.

There are at least a half-dozen similar examples -- including just pursuing random intellectual curiosities. What really helped me come to terms with this is "Structured Procrastination":

http://www.structuredprocrastination.com/

As long as you're doing & not just consuming, you will probably find value.