A lot of comments start off being downvoted for one reason or another and after a few hours end up being upvoted instead. Its nothing to worry about, seems pretty normal on HN and good comments rarely stay downvoted.