Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on I just finished crawling 5.19B web pages, Ask Me Anything. Parent comment
The data does not follow a DFS or BFS pattern so pages/site varies greatly by a host's server capacity and anti-crawling configs.
There was a minimum of 10 seconds between followup requests to the same website unless robots.txt had a lower delay. Pretty polite...
There are no comments on this comment yet.
The data does not follow a DFS or BFS pattern so pages/site varies greatly by a host's server capacity and anti-crawling configs.
There was a minimum of 10 seconds between followup requests to the same website unless robots.txt had a lower delay. Pretty polite...