I always thought the only way downvotes buttons should be implemented is that they're non-functional, in that they do nothing other than let a user take an action; you could perhaps then influence what that specific user sees in the future, to help get an idea of their tastes, or instead not actually affecting where a comment or post shows up in an order.