Can easily get you echo chambers if the numbers are visible and people start using them as agree/disagree buttons.

User moderation via "flag as inappropriate" might work though.