Yeah, it's funny. I find a lot of sedentary sports-enjoyment (particularly fantasy sports) to be way "nerdier" than stereotypically nerdy stuff like D&D.

That said, the sports fans I know really are quite the wide spectrum when it comes to their actual sports/exercise participation. Lot of very active people.