For orthogonality's sake, I would suggest that if most of your posts come from 1:1 conversations, post by post, then perhaps there is a better medium to communicate with that person than through a board like HN.
As an example I give you this discussion. In this thread I have 2 posts, both with 1 karma right now. I do not expect them to go up much if at all. Where would you suggest that I should have gone for this conversation with you?
Here, have an upvote. :)
EDIT: I changed the evaluation to not make the avg karma relative to total karma but instead to the median value of average karma of all users. The downside to this is that Median(AvgKarma) would need to be calculated quite often throughout the day because I doubt pg wants to pay for the resources to run that everytime someone votes.
Whether or not this is a good thing is a matter of preference. (My preference should be obvious.)
Though, truth be told, I've never worried much about karma.