I have recently found a whole lot of value in figuring out what everybody agrees about. A lot of the time that is right. And when it is wrong, it is good to know, and encourages (me at least) to really figure out why the fence is there before I remove it.

Rarely does anyone ever comment to agree on something. And merely being contrarian is not valuable in of itself.

If a few things are over-moderated I might be mildly missing out. So what? Humans are fallible. There is no perfect system. You're not arguing that HN is so moderated that it's completely worthless. You're not even getting close to that.

So what do we do about a system that isn't perfect but is doing a pretty fine job?

That's a weak statement, because it's easy to demonstrate that popular opinion can be misguided.