I share your concerns. I think that this system would be best applied in combination with some sort of flamewar detection. In other words, only enable it in threads that have high comment:upvote ratios, or in threads where there is high velocity back-and-forth or downvoting.