Well, then you're most likely part of minority. And even in that minority I'm willing to bet that a lot of the time is spent in things that have little to do with Computer Science.

Be honest and think of all the time you've spent fiddling with build systems, various random tools (compilers, scripts, databases, version control, etc.).