The difference is that down-voting costs the commenter karma, but the comment stays visible (albeit more grey as the votes go down.) Items or comments that get enough flags go dead and are removed entirely, unless you have "show dead" in your profile.
It's not entirely clear to me the difference between flagging and voting on comments. Spam should be flagged, no question. Comments that are on topic should not be flagged, but if they are information free (such as a witty one-liner with no insight) are usually voted down.
I only flag comments if they are deeply inappropriate, or spam. My position on flagging submissions is as stated above. Most things in which I personally ahve no interest I pass over, but if I think they are not in some way specific to hackers or start-ups, then I'll consider flagging because I think that makes them off-topic.
Not everyone agrees with me, but that's OK.
in any case it sounds like the "HN thing to do" here is for me to advocate my position and see what the community says.