The downvote button (and presumably the ability to downvote) is removed for direct replies to comments you have authored. It is also removed for top level comments on articles you have submitted.