The downvoted commenter would still have no way to tell whether the downvote was related to the reply, since those would most often be coming from different users.