Well there are a couple solutions. One, you could tabulate votes based on the karma of the users. So someone with higher Karma gets bigger weight. That will probably focus the discussionsa and submissions a lot.

Two, you could have interesting comments, based on people voting them up or down, and their replies.

Also you could do some kind of geomtric progression instead of linear, or combine that with the voter's own karma.