This is, frankly, a really annoying aspect of many commenting systems. It's among the reasons I prefer range-based voting (say 1-5 scale) rather than up/down. On the back end it's possible to do some aggregate modeling that works out, for large numbers, somewhat equivalently, but this handles thin data poorly.