Things that are off-topic and illegal aren't the only ones that get downvoted, though. HN is unusual in the sense that it's discouraged to downvote things you dislike; that's the effective status quo in most other communities. In your proposed system, you'll have some people side-voting those comments, and you're going to have others downvoting them.