This would basically be all kinds of drama. You also could not trust the feedback. Some people will down vote for terrible reasons, like a personal grudge. They would not be honest about that.

Also, some down votes are a miss-click. It is now possible to unvote, but that is recent and not everyone will notice if they fat fingered something.

I suggest you give more of your attention to the positive interactions. It is possible to make that choice.