I agree. One idea would be to make cost the downvoter a fraction of their own karma. Perhaps 0.1 points/vote. That's how it works on Stackoverflow and the problem isn't as prevalent there.