I totally agree and have heard that before and thought the same thing.
I've walked through large offices and seen nothing but Lotus Notes on peoples' screens (and I don't mean frivalous emails either, these are 'important' emails/memos).
My experience:
productivity == 1 / count(email + phone calls)
This may be something that is specific to programmers on small teams. I think many of these metrics apply to other areas in business such as Sales where there may be a relation between phone calls or emails and results (Or similarly in customer relations or customer service).A common exception when programming is fixing bugs. When progress is being made there may be a flurry of communication as the source of the bug is identified and fixed.
Don't assume they're implying you're lazy, it could very well be a way of telling you that they don't find it that interesting without being rude. Besides, what's wrong with having different interests? Isn't it normal that one person finds something more interesting than the other, and therefore is willing to spend more of his spare time on it? That doesn't imply one is more lazy than the other, that just means they have different passions.
The second problem is - why not just tell the truth? 'It doesn't really appeal to me, personally.' (talking about neutral subjects here, not requests for favours etc)