Symmetry is symmetry, no matter what I do think of it. If you make symmetrical actions for asymmetrical intents, that is just bad UI.
I've had enough comments downvoted purely on disagreement (including the one above. Or was it "stupid"?) to not to buy your "custom" argument.
The moderation is not symmetrical - it only goes to -4, but has no limit in the other direction.
And it doesn't matter what you think it implies. The reality is that the custom here is not to downmod comments you disagree with. It wasn't always so, even pg at one time thought differently. But experience shows that the discussion is much better when this custom is followed.