Precisely, if I disagree but do not counter a comment, then why should I be allowed to silently hide their comment? It's fine to upvote but downvoting makes no sense, especially anything below 1 discourages contributors, seeing how they're silenced for unpopular opinions.

Agreed. I really found it annoying that Github decided to include -1 on PRs and issues because it encourages people to just pick the cheapest cost option and not express their argument against a change, which is probably one of the best possible ways to block new contributors.