News for hackers. Stuff that matters.

A comment on How companies use fake sites, backdated articles to censor Google results (2017). Parent comment

Re:How companies use fake sites, backdated articles to censo... by askmike on Monday October 01, 2018 @12:47AM (#18110103)
Google crawls the open web non stop. They definitely have this data. It's just not exposed to the (legal) team dealing with DMCA takedown requests..

EDIT: Or as to the example above: whatever application is responsible for picking the main article to show somewhere.

[ Parent | on HN ]
Re:How companies use fake sites, backdated articles to censo... by codetrotter on Monday October 01, 2018 @01:34AM (#18110272)
They crawl the web non-stop yes, but how often an individual site is crawled varies.

> Our crawl process is algorithmic: computer programs determine how often we should crawl each site. If you post new articles on your site throughout the day, you should be able to see them appear on Google News fairly quickly.

> [...]

> From the moment we discover a new article, we'll keep re-crawling it looking for changes. Since we noticed that most changes to articles occur just after they're published, we revisit articles most frequently in the first day after we've found them. After that, we visit them less often.

https://support.google.com/news/publisher-center/answer/4039...

[ Parent | on HN ]
Re:How companies use fake sites, backdated articles to censo... by askmike on Monday October 01, 2018 @02:02AM (#18110340)
Fair point, if your content gets copied to unknown website X which Google hasn't crawled for days they might not be able to tell where the article appeared first.
[ Parent | on HN ]