Why shouldn't that be allowed? I downvote and reply to comments I've downvoted if the comment is worth responding to, ie, I say why I downvoted. I don't downvote to disagree, I downvote because the content of the comment is not constructive or useful to the thread.

In short, I just downvoted you and then replied to you.