For sure, users downweight (and up-weight!) comments. That's the most important factor. But moderators also downweight subthreads, for example if they're off topic. It's a combo of the two.

Same for stories: the ranking is determined by votes, flags, software, and moderation. Users are the most important factor, but alas not a sufficient one. Without moderation, the system gets into unstable states and can't stay within its mandate.

https://news.ycombinator.com/newsguidelines.html