This would inevitably reward first commenters, but that is the situation anyway with current thread voting. I think later commenters as long as not downvoted past a threshold would stay visible with an even(ish) vote count to give those comments the visibility to rise.
What about comments that people disagree with that are still on topic but controversial? Well, we are running out of directions so they should get a side vote. Downvotes to explicitly send to hidden as not relevant (spam, off topic, etc), upvotes for agreeable comments base on popularity, side votes for relevant but disagreeable.
If you have a costume party with 100 people and 10 arrive without costume, they are the outsiders that have to adapt. If 1000 people join your party you're the weird minority and your costume party will be over.
Of course those can also be grounds for disagreement, and I'd be lying if I'd say that I have never been frustrated with a mod when I posted something that was removed even though it clearly didn't break the rules, but I think that kind of false positive is sort of the price you have to pay to keep a community on track.
I'd rather have that than have everything devolve to low quality and effort Threads.