Or it just incentivizes them to click on your profile, click through to comments and down-vote everything not in direct reply.
Arguably that's a higher bar of caring but once that bar is exceeded it's far more damaging to civil discourse because then you've got factually correct technical comments down-voted into oblivion because that person happened to also commit WrongThink(TM) in the comments on an unrelated contentious subject.
You could also argue that it's an incentive to wait a day before taking a minority opinion on a divisive subject but I'm not sure that's something we want to be incentivizing.
I think probably the most important thing is that it clearly communicates what the expected behavior is. I don't think it's immediately obvious that downvoting a response to you is would be considered impolite. (After all, people down-vote things because they happen to disagree all the time; what's the difference if it's in response to something you wrote or in response to something someone else wrote?) But if it's clear that's the social norm, I think most people would just go along with it.
Adding an extra few seconds to retaliation probably does have an effect, but only on people who would consider breaking the social norm anyway.
The other thing is that you don't seem to be able to downvote comments which are a direct reply to your own comment (although you can up-vote them). You can downvote replies-of-replies though.