Since it is activated on Indonesian Wikipedia (I helped them to extend the tool to Indonesian language), I noticed that this tool can hardly capture an obvious vandalism yet. Some other edits are hard to verify whether it is vandalism or not, even by human. I believe this is a work in progress and hope that it will improve over time. But hey, it is the first time a vandalism detection tool is deployed in Indonesian Wikipedia. :)
That's worrisome then. You'd expect a bit more rigor before putting a bot like that into production.
Currently, we're optimizing for AUC since it (1) is hard to cheat -- unlike accuracy or precision/recall individually and (2) it represents the ranking problem well.
See https://en.wikipedia.org/wiki/Receiver_operating_characteris...
"[AUC] is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one"
So, this is useful enough to reduce the workload by 90% right now. As we increase the model's fitness, we should be able to drive that up to 95%, but I doubt we'll get much better than that.
Right. This is really interesting. A while ago I wrote a classifier that predicted based on a number of criteria if a new member of a web community would be trouble or not. It also worked as an aid in classification and not as a stand-alone decision maker. You have a lot more data to work with so that should make the problem a bit more tractable.
Does it use other bits of knowledge about the edit prior probability of a revert for an edit by that particular user or network range for instance?
Any binary 'good or bad' classifier should have at least a couple of stats attached to it to give an indication of how reliable the classifier is.
To complete the circle, it would be nice to know how it performs according to this:
https://en.wikipedia.org/wiki/Precision_and_recall