If anything, I'd suggest the reverse: it should count for more to have more votes from a less tightly coupled portion of the graph. As in, cluster the set of users based on various types of interactions, and include a multiplicative factor in scoring based on the distinctness of the users upvoting something. That way, you can get a modest score by appealing strongly to a small subset of users, but a massive score only by appealing to many distinct sets of users.
Using average karma per comment would be an improvement, but it still favors people who are on the site constantly (so they can be the first to make the obvious comment that everybody upvotes) and frequent the most popular (and lowest effort) subreddits.
Another problem: the last thing reddit needs is more incentive for users to improve their personal karma score. Karma is useful for readers, but that benefit is balanced against the incentives it provides for writers to optimize their karma score above all other considerations. I think HN achieves this balance better than reddit does by deemphasizing personal karma scores in various ways, and this is reflected in the user experience. It's bad enough when karma's just a driver of personal gamification. If it actually provided users with social clout and ability to frame the conversation, well, then it would be something with real world value. You wouldn't need click farms to manufacture viral content, just a few well-placed accounts with high scores. High-karma upvotes as a service. As a side-gig for a careful power user, it would be nigh-undetectable.
Think of the difference between HN and reddit, then multiply it 10-fold. I think that's what you would end up with.
If someone like tptacek (with 259K karma) votes my comment up, I think that's a lot more relevant than someone with 300 karma.