How I trained fake news detection AI with 95% accuracy, and almost went crazy

Re:How I trained fake news detection AI with 95% accuracy, a... by wadkar on Thursday January 11, 2018 @05:38PM (#16128737)

The fakebox doesn’t detect fake news, it detects articles which are factual/real and everything else is labeled as “fake”.

Where’s the dataset? How did you verify the ground truth? Where are the annotation/labeling guidelines?

What’s the definition of factual/real articles? The dataset appears to be created by the author - which isn’t necessarily wrong but to paraphrase Karl Popper (in the context of human knowledge and scientific endeavors):

There are no ‘pure’ facts available; all observations are functions of subjective factors such as interests, expectations, wishes etc.

http://plato.stanford.edu/entries/popper/#GrowHumaKnow

[ Parent | on HN ]

Re:How I trained fake news detection AI with 95% accuracy, a... by saurabhn on Thursday January 11, 2018 @05:52PM (#16128827)

I'm with @wadkar on this. I think the Fake News Challenge Stage 1 (FNC-1) was a good step towards this effort. They acknowledge (almost) all of these concerns and start with Stance Detection as their first stage. In this problem, pairs of article headlines and body text were classified into {Agrees, Disagrees, Discusses, Unrelated}.

Constructively criticism to the OP: I'd suggest they read the nuance and discussions on the Fake News Challenge [0] and then look into their datasets + evaluation code [1] instead of hand-coding their own "biases" into a {"Fake news","Not-Fake-News"} binary classifier. Feel free to replace "Fake News Challenge" with any other similar effort so that OP isn't tasking themselves with the massive task of "Solving Fake News" all alone.

Disclaimer: I don't have any stake in FNC-1

References:

[0] http://www.fakenewschallenge.org/

[1] https://github.com/FakeNewsChallenge/fnc-1

[ Parent | on HN ]

Re:How I trained fake news detection AI with 95% accuracy, a... by wadkar on Thursday January 11, 2018 @06:11PM (#16128950)

Thanks for the FNC links - quite interesting! This would be a nice challenge/dataset for grad students to work as a project in ML/NLP class.

[ Parent | on HN ]