I agree that it makes the site more pleasant, I feel the scores were too often distracting from the actual meat of the comments. Today is a better balance between too much information and too vague: better scores still go up and negative scores are grayed out.

That said, I agree some more context would be useful. Maybe the balance could be adjusted by including some more information. A page could include some overall statistics, for example the top and average score for the page, and/or maybe other useful info that is not hidden but complicated to access, like the top and average karma, etc.