This is a reliable way to identify low value contributions this is less of an issue in a relatively small environment like hacker news where you can in most threads without grave difficulty read all the comments and every story on the front page if you like.

In larger scoped things you will have much more content than anyone could possibly consume and most of it is bad. Keeping score is a way of avoiding everyone having to go through the same pile of drek. Keeping score is only half as effective if you can only up vote. Consider two tallies of several contributions. First two bad contribution neither sufficiently bad to be worthy of being banned.

Contribution A: Imagine 100 people viewed, 10 people liked it, 70 people hated it 20 were indifferent.

With down votes: -60 Without: 10

Contribution B: 10 people liked it, 10 hated it, 80 indifferent.

With down votes: 0 Without: 10

Now consider a positive contribution C with fewer votes 10 for and 1 against.

With down votes: 9 Without: 10

Now something outrageous that attracted a ton of engagement based on how bad it is. Maybe they said we should raise corgis and kitties as meat animals! 980 votes to the negative and 20 to the positive.

With down votes: -960 without: 20

Without down votes A B and C are identically scored despite B being worse than C and A being much worse. Even worse D despite being horrible is scored the highest!

Not applying the karma of a comment to the commenter has similar downsides. You would be throwing away good useful information your users have given you.