News for hackers. Stuff that matters.

xagarth

xagarth on Hacker News

Re:Build a crawler to crawl million pages with only one machine in jus... on Thursday March 02, 2017 @03:45PM (#13778023)
Crawling described here is very inefficient. For efficient and high performance crawling I recommend libcurl and curlmulti.