News for hackers. Stuff that matters.

A comment on I just finished crawling 5.19B web pages, Ask Me Anything.

Re:I just finished crawling 5.19B web pages, Ask Me Anything by tlack on Wednesday April 19, 2017 @07:21PM (#14152671)
what did you do to avoid winding up in endless GET url loops? How deep did you get per site, and how did you schedule followup requests?
[ Parent | on HN ]
Re:I just finished crawling 5.19B web pages, Ask Me Anything by dor_jack_2 on Wednesday April 19, 2017 @07:43PM (#14152778)
Loop/spam prevention was done by mixnode, I'm not sure how they do it.

The data does not follow a DFS or BFS pattern so pages/site varies greatly by a host's server capacity and anti-crawling configs.

There was a minimum of 10 seconds between followup requests to the same website unless robots.txt had a lower delay. Pretty polite...

[ Parent | on HN ]