News for hackers. Stuff that matters.

A comment on Build a crawler to crawl million pages with only one machine in just 2 hours.

Re:Build a crawler to crawl million pages with only one mach... by xagarth on Thursday March 02, 2017 @03:45PM (#13778023)
Crawling described here is very inefficient. For efficient and high performance crawling I recommend libcurl and curlmulti.
[ Parent | on HN ]

There are no comments on this comment yet.