It's an interesting idea, but it would systemically tendency towards ossifying views that the community already holds/has held in the past. If those who hold a certain opinion already have more karma, a feedback loop is created whereby they're more free to downvote the opposing opinion, the other side is less able to downvote theirs, so the original group doesn't lose as much karma, etc etc.