I've never understood that. It might be because people are afraid of retaliatory downvotes in return. I have enough rep there that I don't worry about it, but even so it never seems to happen.

Two thoughts.

There should be an auto block feature. If you have it on and they downvote you, they get blocked and can't see your posts anymore.

Second thought about gamification. A downvote should cost the person a point or two of their own karma.