News for hackers. Stuff that matters.

A comment on Build a crawler to crawl million pages with only one machine in just 2 hours.

Re:Build a crawler to crawl million pages with only one mach... by mfontani on Tuesday February 28, 2017 @11:38AM (#13754991)
You might want to set a specific user-agent for your crawler
[ Parent | on HN ]

There are no comments on this comment yet.