Doesn't every comment start with 1 point, though? People trying to game the system could still do so simply by commenting a lot (maybe on older, less visible submissions that won't get them down-votes). Maybe every comment should start at 0 points?
My point is if you remove karma for submitted stories you only get people who earned the right to downvote via well thought out comments.
My hypothesis is that these are the type of people who are more likely to counter a point they disagree with, with a well thought out comment, rather than a downvote, as that's how they got the karma they have to begin with.