I agree with you, my opposition is actually regarding posts. I have been active on a number of posts over the last couple of years that were very much on topic but were on controversial issues, and they will get flagged to death after being on the front page for a few minutes, and at that point further conversation is dead and the post sinks into irrelevance.

The fallacy here: when you're only exposed to stuff that everyone agrees about, you're missing out.

I have recently found a whole lot of value in figuring out what everybody agrees about. A lot of the time that is right. And when it is wrong, it is good to know, and encourages (me at least) to really figure out why the fence is there before I remove it.

Rarely does anyone ever comment to agree on something. And merely being contrarian is not valuable in of itself.

If a few things are over-moderated I might be mildly missing out. So what? Humans are fallible. There is no perfect system. You're not arguing that HN is so moderated that it's completely worthless. You're not even getting close to that.

So what do we do about a system that isn't perfect but is doing a pretty fine job?

That's a weak statement, because it's easy to demonstrate that popular opinion can be misguided.