News for hackers. Stuff that matters.
Re:Ask HN: What are your news sources other than HN? by pjc50 on Tuesday December 31, 2019 @09:15AM (#21921102)
> I am using some fairly simple word analysis techniques (most of the effort is scraping the data from these sites and deciding what constitutes a real news story) and most of the language on the BBC and the Guardian are similar. I suspect that most of them just copy and paste stories tbh.

There is a more serious underlying problem you've spotted, which is "jounalism as stenography". Lots of people can put out press releases, which then really do get copy-pasted into articles. Same with "wire copy" sourced from AP or Reuters. This can be abused by bad actors to get obvious or difficult-to-refute falsehoods into the public memetic consciousness by repeating them.

Re:Ask HN: What are your news sources other than HN? by uncle_j on Tuesday December 31, 2019 @09:57AM (#21921350)
> This can be abused by bad actors to get obvious or difficult-to-refute falsehoods into the public memetic consciousness by repeating them.

This is exactly why I started the project also I know there are hit pieces that are coordinated between the news-sites. I wonder how much other news that is reported is also coordinated reporting. However I have no direct proof of this and I cannot start from trying to prove something that I suspect. First I need to collect and analyse data fairly.

So far I've just used newspaper to do the hard work of scraping for me (https://github.com/codelucas/newspaper) some basic filtering techniques e.g. I discard everything that has a question mark at the end of it (Betteridge's law of headlines and an MySQL to store articles etc. Then I run the second script to do some basic string analysis. I am going to be building a small web app to display the results.

So far it is early days (It about 4 or 5 python files and a config file) and I have got two other projects I am working on that probably need completing first.