Re:AI service gives Wikipedians ‘X-ray specs’ to see through...
by jacquesm on Wednesday December 02, 2015 @02:33PM
(#10665242)
Thank you. 75%, that's not bad but it's also not exactly 'x-ray eyes' grade yet. This is a pretty tough problem you're tackling, I can see now why it is not a bot but an augmentation device. It would be nice to track the accuracy over time, to see if the vandalizers wise up to how the bot makes the decisions to game it.
[ Parent
| on HN ]
Re:AI service gives Wikipedians ‘X-ray specs’ to see through...
by halfak_wmf on Wednesday December 02, 2015 @02:56PM
(#10665397)
Actually our AUC is 90% and that beats the state of the art in the literature. Accuracy is easy to game. We could get 97.5% accuracy by just predicting False all of the time. We don't advise reviewers to use the prediction directly, but rather to set thresholds on the probability. 70% probability is a very conservative filter and will catch about 10% and a little bit less than half of them will actually be damaging.
So, this is useful enough to reduce the workload by 90% right now. As we increase the model's fitness, we should be able to drive that up to 95%, but I doubt we'll get much better than that.
Re:AI service gives Wikipedians ‘X-ray specs’ to see through...
by jacquesm on Wednesday December 02, 2015 @03:10PM
(#10665470)
> 70% probability is a very conservative filter and will catch about 10% and a little bit less than half of them will actually be damaging.
Right. This is really interesting. A while ago I wrote a classifier that predicted based on a number of criteria if a new member of a web community would be trouble or not. It also worked as an aid in classification and not as a stand-alone decision maker. You have a lot more data to work with so that should make the problem a bit more tractable.
Does it use other bits of knowledge about the edit prior probability of a revert for an edit by that particular user or network range for instance?