For comments, I usually only flag when people are egregiously violating site guidelines or are just being dicks in general, and I think most other potential flaggers often agree with my flags in those cases, and pile on. If I just think that a comment doesn't add anything substantive, I'll merely downvote.
I'll also downvote if I strongly disagree with someone's position, after reading an older post on HN from pg saying that's an accepted use for downvotes[0]. I'm still somewhat torn on this, as I'm not sure a nearly 15-year-old post on policy (from someone who seems not to be much involved with HN these days) should still be considered authoritative, as community norms change over time. I also don't love that using downvotes to express disagreement also has the effect of hiding contrary opinions, which can often hold a lot of value. So I try to reserve my disagreement downvotes for cases where I believe I have evidence the poster is objectively wrong about what they're saying.
My 3 pages of flagged submissions seems a bit more random. Perhaps I don't remember the reason just skimming through, but some of them are certainly accidental.