Most people don't vote. But I agree that downvoting can be an emotionally charged action - completely devoid of reason.

I tend enjoy downvotes that have no reason because they validate a touched nerve, but some will give a reason as to why they downvoted and open a dialogue. Thats a gold feature of HN I think.