saurabhn

I'm with @wadkar on this. I think the Fake News Challenge Stage 1 (FNC-1) was a good step towards this effort. They acknowledge (almost) all of these concerns and start with Stance Detection as their first stage. In this problem, pairs of article headlines and body text were classified into {Agrees, Disagrees, Discusses, Unrelated}.

Constructively criticism to the OP: I'd suggest they read the nuance and discussions on the Fake News Challenge [0] and then look into their datasets + evaluation code [1] instead of hand-coding their own "biases" into a {"Fake news","Not-Fake-News"} binary classifier. Feel free to replace "Fake News Challenge" with any other similar effort so that OP isn't tasking themselves with the massive task of "Solving Fake News" all alone.

Disclaimer: I don't have any stake in FNC-1

References:

[0] http://www.fakenewschallenge.org/

[1] https://github.com/FakeNewsChallenge/fnc-1

To complete the picture, you might've been taught a completely different mnemonic in other countries. The one I was taught was BODMAS ( "B"rackets, "O"f or "O"rder, "D"ivision, "M"ultiplication, "A"ddition and "S"ubtraction) and some African countries teach this as BIDMAS [0]. Caused a lot of confusion when I came to the US and drew a blank when someone mentioned PEMDAS in an example.

[0]: https://en.wikipedia.org/wiki/Order_of_operations#Mnemonics

What checks does the Wiki model offer against, say, a 4chan-style brigading? I love the idea, I just want it to be bulletproof too.

Sorry, just saw this after all these days. Do check out Adversarial Machine Learning as a really cool next step.

That's really cool! Quick, unsolicited 2¢:

1. Your data is severely imbalanced, so accuracy is a very misleading metric to use here. From what I see, you have a 1:20 imbalance (malicious vs non-malicious distribution). This affects both the metrics and induces bias in classification.

2. I'd like to add to the other comment asking you for calibration curves and see what your minority class performance looks like in terms of precision, recall, f-beta, average precision (area under precision-recall curve).

3. Then, try and see if resampling helps or hurts the predictive performance- it typically speaks to the level of noise and small disjuncts in the data.

4. I see you've done a 0.2 split for test-train, but try and eliminate split bias by using stratified cross validation. This would ensure that you didn't just get lucky with random seed = 42 and get a really great test set.

All of these can be implemented using sklearn and imbalanced-learn [0]. Not included- deeper dive into cost sensitive and adversarial techniques. Let me know if you have any more questions and keep up the good work!

[0]: https://github.com/scikit-learn-contrib/imbalanced-learn

Source: PhD in imbalanced machine learning.

This is a super interesting topic to work on! There's an academic initiative which does exactly this, it's called the Digital Historical Architectural Research and Material Analysis or DHARMA [0] for short (it's at the University of Notre Dame). Figuro3D [1] is a startup (also out of Notre Dame) which borrows a lot from DHARMA and applies it to 3D printing.

[0]: http://architecture.nd.edu/research-publications/dharma/

[1]: http://www.figuro3d.com/

Edit: formatting