I'm fine with the current system, and it seems to work quite well most of the time. I kinda feel like "works quite well most of the time" is about all I can hope for on most things. It usually buries "bad" (yes, that's a subjective word) comments here. It's not perfect, I've seen perfectly reasonable things get grayed out sometimes, but these things happen. It's not a perfect system, but it usually gets the job done.

That being said, I do think your criticisms are good, but I just don't think those are reasons to get rid of downvoting here on HN.