News for hackers. Stuff that matters.

A comment on Ask HN: Has anyone ever crawled over a billion pages? How much did it cost?. Parent comment

Re:Ask HN: Has anyone ever crawled over a billion pages? How... by usgroup on Wednesday September 14, 2016 @06:18AM (#12495460)
Just in case you don't know common-crawl makes available a huge crawl dataset
[ Parent | on HN ]
Re:Ask HN: Has anyone ever crawled over a billion pages? How... by outpan on Wednesday September 14, 2016 @01:27PM (#12499252)
Common Crawl is great! however, some use cases require larger crawls with a higher frequency.
[ Parent | on HN ]