News for hackers. Stuff that matters.

Ask HN: Scientific Study of HN flag dynamics

Would there be any interest in a scientific study of the effectiveness of HN flagging, including:

  - false positives
  - false negatives
  - brigading
  - flagged topics vs off-HN contexts
  - flagged topic trends over extended time
  - content origin (domain, geo, org, author)
Pointers to prior research on distributed flagging and moderation would be welcome.
( 15 comments | 7 points | on HN )
Re:Ask HN: Scientific Study of HN flag dynamics by walterbell on Monday February 28, 2022 @01:39AM (#30497025)
Response to aaron695: https://news.ycombinator.com/item?id=30496522

> The open data would tell an incomplete picture. It would still be an interesting to see what you can pull out, but you'd be guessing why a lot of the time.

The current implementation is like the DMCA. Anything can be censored for any reason without input from HN moderators. If the submitter is aware of the process, they can appeal to HN moderators. If HN readers saw the submission before it was flagged, they can also appeal to HN moderators.

If there were organized efforts to censor specific topics, one mitigation could be to deny the benefits of censorship. This could be done from public data by extracting a queue of flagged submissions and feeding them into another level of non-public analysis, automated + human. The surviving stories could then be posted to a dedicated site, e.g. using https://github.com/lobsters/lobsters.

If there were organized efforts to censor specific topics, a secondary public log of valid-but-censored articles could trigger the Streisand effect. Not only could this deny the benefit of censorship, it could cause censorship to have the opposite effect, reducing demand for organized flagging. This would have no impact on organic, non-coordinated flagging.

Re:Ask HN: Scientific Study of HN flag dynamics by WalterGR on Monday February 28, 2022 @03:56AM (#30497823)
Wait - are you proposing to study flagging or “censorship” on HN?

Flagging certainly exists. What is your evidence in favor of censorship existing here?

If memory serves, there are website(s) that track which submission get flagged. If you’re concerned about a favored view or topic being flagged frequently, you could start there.

Otherwise you risk exerting lots of effort to simply find out that a favored view or topic isn’t shared by lots of people here. And, well, not all of them are equally important, or equally important to the HN audience.

Re:Ask HN: Scientific Study of HN flag dynamics by walterbell on Monday February 28, 2022 @04:18AM (#30497947)
The study would be on flagging, which is typically done by independent HN users.

HN has non-public mechanisms to detect some coordinated activity (e.g. upvoting submissions) across registered usernames.

> If there were organized efforts to censor specific topics

Note the "if" at the beginning of the sentence. It's not about transient topics, but long-term patterns. Broadly speaking, "censorship" is only one possible motive for coordinated activity, flagging or otherwise.

> If memory serves, there are website(s) that track which submission get flagged.

Thanks for the pointer, found https://news.ycombinator.com/item?id=29144989 & https://news.ycombinator.com/item?id=29113079 & https://github.com/MoserMichael/flagged-hn.

Re:Ask HN: Scientific Study of HN flag dynamics by WalterGR on Sunday February 27, 2022 @10:16PM (#30495963)
Definitely.

Add to the facets: the submitted domain. Maybe posts by the armchair experts and quarterbacks on Substack get flagged more often than the WSJ, for example.

Re:Ask HN: Scientific Study of HN flag dynamics by walterbell on Sunday February 27, 2022 @10:19PM (#30495976)
Thanks, added "content origin".
Re:Ask HN: Scientific Study of HN flag dynamics by musawo on Monday February 28, 2022 @03:04AM (#30497508)
Re:Ask HN: Scientific Study of HN flag dynamics by walterbell on Monday February 28, 2022 @03:27AM (#30497644)
An larger and noisier category for study: recurring patterns of low-quality (downvoted, flagged) spam comments on some topics.
Re:Ask HN: Scientific Study of HN flag dynamics by brudgers on Monday February 28, 2022 @01:21AM (#30496948)
There are no objective criteria for classifying flags as false positive/negative.
Re:Ask HN: Scientific Study of HN flag dynamics by walterbell on Monday February 28, 2022 @01:41AM (#30497030)

  What does [flagged] mean?

  Users flagged the post as breaking the guidelines or
  otherwise not belonging on HN.

  Moderators sometimes also add [flagged] (though not usually 
  on submissions), and sometimes turn flags off when they are 
  unfair.
A false positive is a user flag that was reversed on appeal, after manual review by HN human moderators.

In a study, a list of false positive flags could be derived from public data.

In a study, additional false positive candidates could be identified after manual review.

Re:Ask HN: Scientific Study of HN flag dynamics by brudgers on Monday February 28, 2022 @02:35AM (#30497333)
Flags add gravity. Sometimes, but only rarely, to the point of flagged-to-dead. Mostly flags just move submissions away from the top of the front page.

It is very rare for those to be “deflagged.” Rare enough that there’s a moderation comment about turning flags off.

And an appeal that the community be on its better behavior. They are editorial exceptions not rule following corrections.

Re:Ask HN: Scientific Study of HN flag dynamics by walterbell on Monday February 28, 2022 @02:50AM (#30497419)
> Flags add gravity.

For the purpose of a study, gravity can be inferred from story rank.

> Sometimes, but only rarely, to the point of flagged-to-dead.

A study could examine the three categories of:

  - gravity
  - dead
  - unfair
It would need to be long enough that all of those categories were non-zero.
Re:Ask HN: Scientific Study of HN flag dynamics by brudgers on Monday February 28, 2022 @03:33AM (#30497679)
There’s no unfair…or fair.

HN’s not a court of law.

Re:Ask HN: Scientific Study of HN flag dynamics by walterbell on Monday February 28, 2022 @04:15AM (#30497923)
This sub-thread is about false positives, which has already been stated as a user-flagged event that was subsequently reversed by an HN moderator. That is a physically observable event that is visible in the HN public record, therefore it is an event that is subject to scientific study. It either exists or does not, so it is objective.

In this sub-thread, "unfair" is quoted directly from the HN guidelines as the term for the objective event of a reversed flag, i.e. a "false positive", which we are discussing. It's not being used here as an adjective, it's being used to connect this discussion to the official HN FAQ.

Re:Ask HN: Scientific Study of HN flag dynamics by sjg007 on Monday February 28, 2022 @01:30AM (#30496992)
Is this data publicly available?
Re:Ask HN: Scientific Study of HN flag dynamics by walterbell on Monday February 28, 2022 @01:45AM (#30497062)
Raw flag data is not public. Some studies are possible on the public data.

Any interesting results could be investigated further, privately, by HN moderators.