Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on How to Crawl the Web Politely with Scrapy. Parent comment
http://www.bing.com/search?q=%22We+have+been+receiving+a+lar...
I agree that crawl-delay is rare, and often it's set too long so that it's impossible to fully crawl a site -- as if the webmaster set it up 10 years ago and never updated it as their site got faster and bigger.
There are no comments on this comment yet.
http://www.bing.com/search?q=%22We+have+been+receiving+a+lar...
I agree that crawl-delay is rare, and often it's set too long so that it's impossible to fully crawl a site -- as if the webmaster set it up 10 years ago and never updated it as their site got faster and bigger.