Currently, we're optimizing for AUC since it (1) is hard to cheat -- unlike accuracy or precision/recall individually and (2) it represents the ranking problem well.
See https://en.wikipedia.org/wiki/Receiver_operating_characteris...
"[AUC] is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one"
So, this is useful enough to reduce the workload by 90% right now. As we increase the model's fitness, we should be able to drive that up to 95%, but I doubt we'll get much better than that.
Right. This is really interesting. A while ago I wrote a classifier that predicted based on a number of criteria if a new member of a web community would be trouble or not. It also worked as an aid in classification and not as a stand-alone decision maker. You have a lot more data to work with so that should make the problem a bit more tractable.
Does it use other bits of knowledge about the edit prior probability of a revert for an edit by that particular user or network range for instance?
That's worrisome then. You'd expect a bit more rigor before putting a bot like that into production.