I have a suspicion that upvoting direct replies increases the overall standing of my post (ranking seems to aggregate positivity), so I do it with extreme prejudice -- I try to only upvote good-faith comments that disagree with or correct me.