News for hackers. Stuff that matters.

A comment on Build a crawler to crawl million pages with only one machine in just 2 hours.

Re:Build a crawler to crawl million pages with only one mach... by drallison on Wednesday March 01, 2017 @11:46AM (#13765334)
Another worthwhile article if you are building a crawler. http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-bil...
[ Parent | on HN ]
Re:Build a crawler to crawl million pages with only one mach... by plantpark on Wednesday March 01, 2017 @09:09PM (#13770814)
I've seen this article.Great article about distributed crawler. Mine just is a demo version of his.
[ Parent | on HN ]