You know, I could have sworn I read in the guidelines somewhere that downvotes aren't to be used just to signal that you disagree, but in diving a bit deeper I think that's just one of those things that people say, but doesn't actually ground to any specific rules/guidelines.

And in fact, an early post says the opposite -- downvoting because you disagree is fine: https://news.ycombinator.com/item?id=117171

You might be thinking of the Reddit guidelines, where this rule is famously never observed.