Do you have evidence for this? It doesn't change the publicly displayed number, but I was under the impression that it does change the ordering. That did seem to be the case last time I experimented with this. On a video with only a dozen or so comments I downvoted a comment with no upvotes, and then opened the video page in an incognito tab, and the comment I downvoted was right at the bottom. Could have been a coincidence, but the probability of that seemed low at the time.