If you rarely downvote and others allways downvote items they don't like, your opinion is going to be underrepresented. I think that is a harm. As I said elsewhere, downvotes are too ambiguous in meaning.