Even simpler solution: present a downvote button, make it show a "downvoted" label on the item to the user who pressed it, don't have it change the actual score at all.

In other words, give them a placebo button: https://en.wikipedia.org/wiki/Placebo_button