News for hackers. Stuff that matters.

halfak_wmf

Hey folks. I just put this dataset together. It uses a prediction model that we've been working on for a while. This helps address some of the propensity modeling issues that past researchers have looked at. This dataset should make a lot of research and analysis a lot easier.
Actually our AUC is 90% and that beats the state of the art in the literature. Accuracy is easy to game. We could get 97.5% accuracy by just predicting False all of the time. We don't advise reviewers to use the prediction directly, but rather to set thresholds on the probability. 70% probability is a very conservative filter and will catch about 10% and a little bit less than half of them will actually be damaging.

So, this is useful enough to reduce the workload by 90% right now. As we increase the model's fitness, we should be able to drive that up to 95%, but I doubt we'll get much better than that.

Sorry about the formatting issues. I can't seem to edit the post :/
+1 for Egalitarianism. So, when I use the word "feminism", I'm referring to the academic field that explores power structures in social systems. These power structures go far beyond gender issues despite what the label "feminism" implies. It's this broader sense of justice/morality in social spaces that I would like to engage with and draw inspiration from.

There are two core bits of feminist theory that I draw from as inspiration in organizing our work on this system:

* Empowerment vs. Power over -- I'm a computer scientist and I've been studying quality control in Wikipedia for 10 years. That makes me very powerful in this conversation. I could use that power to browbeat Wikipedians into changing their quality control practices to what I think we should do. But on the other hand, I don't think that my perspective is absolute. A lot of Wikipedians know better than I do how they should curate the encyclopedia. So instead, I'm trying to empower Wikipedians and our community of tool developers to imagine how they'd like quality control to work. My research[1,2,3] suggests that we need reform here and I'd rather see that reform through innovation than command and control. Honestly though, I don't need to look to feminism for this insight. I came to the same conclusion after considering Wikipedia as a complex social system[4].

* The potential for systemic biases -- As other comments have pointed out, there's inevitably systemic bias in people's behavior as they evaluate good and bad edits in Wikipedia because these judgements are inherently subjective and there's a strong incentive to push back against anything that is new and unusual -- especially, things that don't follow the western scholarly traditions of reference information. Once encoded into a black box algorithm, these biases can become even more hidden and substantially more impactful. One bias that is plainly obvious at the moment is against anonymous editors. While a lot of vandalism may come from anonymous editors, so does ~25% of productive contributions to Wikipedia articles (it's my job to measure these things). We're actively working to nerf the "is anonymous" feature and find better sources of signal.

1. Halfaker, A., Geiger, R. S., Morgan, J. T., & Riedl, J. (2012). The rise and decline of an open collaboration system: How Wikipedia’s reaction to popularity is causing its decline. American Behavioral Scientist, 0002764212469365. http://www-users.cs.umn.edu/~halfak/publications/The_Rise_an... 2. Halfaker, A., Geiger, R. S., & Terveen, L. G. (2014, April). Snuggle: Designing for efficient socialization and ideological critique. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 311-320). ACM. http://www-users.cs.umn.edu/~halfak/publications/Snuggle/hal... 3. Halfaker, A., Kittur, A., Kraut, R., & Riedl, J. (2009, October). A jury of your peers: quality, experience and ownership in Wikipedia. In Proceedings of the 5th International Symposium on Wikis and Open Collaboration (p. 15). ACM. http://www-users.cs.umn.edu/~halfak/publications/A_Jury_of_Y... 4. Halfaker, A., Kittur, A., & Riedl, J. (2011, October). Don't bite the newbies: how reverts affect the quantity and quality of Wikipedia work. In Proceedings of the 7th international symposium on wikis and open collaboration (pp. 163-172). ACM. http://www-users.cs.umn.edu/~halfak/publications/Don't_Bite_... 5. Geiger, R. S., & Halfaker, A. (2013, August). When the levee breaks: without bots, what happens to Wikipedia's quality control processes?. In Proceedings of the 9th International Symposium on Open Collaboration (p. 6). ACM. http://www-users.cs.umn.edu/~halfak/publications/When_the_Le... 6. Halfaker, A., & Riedl, J. (2012). Bots and cyborgs: Wikipedia's immune system. Computer, (3), 79-82. http://stuartgeiger.com/bots-cyborgs-halfaker.pdf

Actually, we openly report a wide variety of statistics about our models. E.g. if you go to https://ores.wmflabs.org/scores/enwiki, the service lists out the models that are available for English Wikipedia along with test statistics. You can also browse our test statistics in our documentation. See https://meta.wikimedia.org/wiki/ORES/damaging#English_Wikipe...

Currently, we're optimizing for AUC since it (1) is hard to cheat -- unlike accuracy or precision/recall individually and (2) it represents the ranking problem well.

See https://en.wikipedia.org/wiki/Receiver_operating_characteris...

"[AUC] is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one"

Yeah. We're actively working with patrollers to chase down mistakes and retrain the models. This is a system that will require continual improvement. Luckily, there's a lot of researchers working on this problem so we're drawing from a lot of brilliance beyond our little team at the Wikimedia Foundation.